git-annex can use the WWW as a special remote, downloading urls to files. See using the web as a special remote for usage examples.
notes
Currently git-annex only supports downloading content from the web; it cannot upload to it or remove content.
This special remote uses urls on the web as the source for content. git-annex can also download content from a normal git remote, accessible by http.
@spiderbit, to support recursively adding content, it would need to parse html, and that's simply too complex and too unlikely to be useful to many people.
Sorry, should have read the man page.
of course I have to use %url and %file
So it works with "rsync %url %file" but it seems to not work recursive also it renames the files instead of adding them with their normal name. So not useful for what I want to do.
I want to access a normal unmodified directory on my server and add the files to my local directory. That would be a minimal setup, everything else means just extremly big setups with assistant runnig and a cronjob to delete unused files, and lots of cpu load for indexing this files on the server.
I think such minimal setup would be great to get startet without having very complex setups, you dont want to commit to such a tool and hours of setup to get somethnig useful, just to look if its useful for you.
there are 2 aproaches either have a normal repos on the server, with again cronjob and flat setup. which is quite a setup to get.
or use only a real repos on the client, both have big disadvantages normal repos complex setup, to complex to just test it. web links seem to be simple enough but is not recursive therefor only good for youtube links or stuff like that.
Is there really no simple solution for what I want to do?
When it says "arbitrary urls", it means it. The only requirement is that the url be well formed and that wget or whatever command you have it configured to use via annex.web-download-command knows how to download it.
Update 2018: That used to be the case, but it's now limited by default to http and https urls.