2 July 2010 at 18:25 #19129
Some users have commented on the download times for SPT on Box.net. For my home and office connections, the download times seem comparable with other organisations’ servers, i.e. about half an hour for the Electricity and Energy 400MB file at the office, or about two hours at home.
Do you experience much faster downloads from other organisations? (Personally, I don’t, and Box.net is a large filesharing company.)
Should we take advantage of software such as BitTorrent (which has wholly legitimate uses, beyond its widespread use in illegal file sharing)?
Any other thoughts on download speeds?
2 July 2010 at 19:45 #19133
I quite like the general concept of peer to peer file sharing things like BitTorrent. It’s very Clay Shirky, isn’t it? I have little knowledge of how it would work with the 2000-odd current holders of SPT materials, though. Would enough people have their computers turned on and allow access to the zip files to make it advantageous?
I suppose there’s bound to always be the original ‘seed’ open, possibly still hosted by something like box.net if they allow that sort of thing, and then anything else is just a bonus, really.
3 July 2010 at 18:02 #19134
Jon, I am seeing 3-4 hours to download Electricity and Energy on my connection at home, and that isn’t right at all for a good file sharing service. I should be able to download that file in less than 30 minutes usually. It does look like box.net is very slow and it is a Saturday afternoon (whether that is good or bad I don’t know). I use a different file sharing service and have no problems. David.
4 July 2010 at 16:12 #19135
As posted in ‘resource’ area, my download was about 30-40 kbps, so about 3-4 hrs in total. This is considerably slower than most other files I’ve downloaded in the past.
As for Torrents, I would gladly seed such resource torrents if I knew that people needed them…
4 July 2010 at 16:50 #19136
Today (Sunday) I have just downloaded the same 400MB file in 25 minutes from home. Perhaps we are seeing congestion due to the files being hosted in the US? This is quite common because international bandwidth is more subject to congestion that bandwidth within the UK. I am wondering why we are not hosting the files in the UK or the EU so that we get better download times? talkphysics.org is in the UK, so why not the files?
6 July 2010 at 12:21 #19137
I think local UK hosting is probably a better answer than BitTorrent. BitTorrent is difficult to use, and I assume is banned in many schools.
6 July 2010 at 18:54 #19138
Well, box.net is banned in my school.
7 July 2010 at 12:10 #19139
Edwin makes a point that I was thinking… BitTorrent is not going to be easy for some people, it needs to be “click and download”.
The point that griff makes also emphasises why we really need the files shared locally in the UK and on the iop.org domain name so that they can be accessible from schools. Don’t Bristol have their own servers, or did I get that wrong when I visited once and someone said they had?
8 July 2010 at 20:34 #19140
Excellent feedback – thank you.
Ok – some sites, such as box.net, will always be banned. If we move to another commercial UK file sharer (or BitTorrent), we’d probably find that banned by default. I think we each have to fight our internal school battles to show IT managers that they need to allow educational uses.
However, I’m persauded that there may be a problem with US hosting of large files. I’ll try to find a way of testing this – I’m unlikely to be at home on enough different times of different days to repeatedly download the file. Any other suggestions on how to test internet congestion? Is there a map or log anywhere (I couldn’t find anything)? I could ping repeatedly, but again it depends only on my own connection at the times I’m available to try.
I will explore the hosting available from within IOP Publishing (our Bristol arm hosts most of our web servers). Box.net came about because it offered the services that were needed at the time – we haven’t had a problem with congestion before.
Do bear in mind that all of this is evolving relatively quickly – the first attempt at hosting SPT online came less than 12 months ago, so this is a considerably more advanced discussion than was taking place 12 months ago!
12 July 2010 at 11:30 #19141
Hi – I’ve just broken the EE SPT topic into four files of roughly 100MB each, and will post them on Box.net. That might help with downloading, particularly with variable connection speeds.
There doesn’t seem to be any underlying problem with Box.net (given that many users can download quickly from them), and our IT staff feel it’s unlikely to be an issue with Transatlantic bandwidth. Please bear in mind that individual connections might vary.
I’ll keep an eye out for future user comments, in case we build stronger evidence that there is a problem with Box.net. Thank you.
12 July 2010 at 14:59 #19142
The 100MB chunks may help.
The fact that individual connections may still vary (depending on ISP) is another argument for hosting in the UK. Most major ISPs peer at LINX in London (and other similar facilities) this helps keep traffic inside the UK and speeds up data transfer. So whilst there may be no underlying problem with box.net or the transatlantic bandwidth in general, it doesn’t remove the need to host the files in the best possible way for end users (even if any problems are not directly related to the infrastructure we choose to use). Anyway, it is more environmentally responsible to host nearest to your target audience
12 July 2010 at 15:17 #19143
I am concerned about switching provider without strong evidence that the US location is the root cause.
However, can you suggest a UK file host so that we can consider it? The IOP doesn’t provide a file-hosting facility, and our publishing arm (IOPP) uses another US company.
16 July 2010 at 12:51 #19144
David emailed me:
> Do IOP pay box.net for the file sharing facility?
We pay a few hundred pounds per year for their Enterprise service (I don’t want to simply post a contract price here).
> There are plenty of other ways to do this including hosting the files on a simple UK based server and uploading the files via FTP. I can recommend a company in Norway (which is 2 hops from LINX) who I have extremely close ties with, and they might be interested in doing a deal for hosting the files. However, UK storage would work best, and then Europe, and then the US in that order. I must admit I don’t understand why IOP can’t do it…
The internal charge requested by IOPP (our publishing arm) for the service put it outside our project budget.
> Let me know if you are interested.
Box.net does add some extra functionality beyond a “bare” FTP server, providing an easy user interface (e.g. our Physics Network Co-ordinators are part-time contractors across the British Isles who many not familiar with FTP or be easily available for IT training), allowing us to nominate collaborators, structure the filespace easily, tag the files, etc.
We’d be interested if:
* we could show that download times were significantly faster for a significant number of users (this was what started the ball rolling!),
* it was a similar (or preferably cheaper…) price,
* it was similarly easy to use, including transferring our 165 files (and growing…) with their structured filespace, totalling 2.6GB.
Thank you for the ideas. I’m finding the interactive part of the Web 2.0 interesting – a great way to harness everyone’s knowledge.
16 July 2010 at 13:21 #19145
This is probably me being thick, or just not computery enough, but why can’t the same server do both?
Are there services out there that run forums and do file hosting?
It sounded on another posting that you might look to revamp facilities. Might this just be an extra thing to put on your shopping list?
It would, for instance, save quite a lot of that ‘please don’t publish this box.net link elsewhere’ stuff that’s understandably up at the moment. You could just ‘click to download’ and anyone who could get to the talkphysics site could get to the files.
There is bound to be a sound technical reason why this sort of thing doesn’t happen, but do feel free to educate me.
16 July 2010 at 14:02 #19146
griff – the same service can do both as you say. Users upload files to be shared on talkphysics.org already don’t they?
Jon – an FTP server can have a simple web-based file management system that non-FTP users can make use of for uploading files etc… they are not difficult to install, and most commercial file hosting would include them. What specific features do you need that box.net provides?
16 July 2010 at 14:27 #19147
I am sorry, but the IoPP, where TalkPhysics is hosted, has security installed that would defeat hackers that could get into Fort Knox. It is simply not the case that one can get both systems (fileserver and social network) easily onto the same server in this case. TalkPhysics is on a heavily isolated VPS, hosted on IoPP. Please don’t think we did not try to persuade them to do both, as this would have made our lives simpler. But the price they quoted for file hosting was prohibitive. We simply do not have the budget for what they were asking.
Elgg does not provide for easy file management. You can verify this for yourself by uploading a decent sized file and then trying to replace it with a newer version.
We did charge the then PM (who had form in the matter of It projects within the IoP) with finding an ftp service that could be used without technical training – they are far and few between.
AS you can imagine, trying to guess what sites are not banned in every school is a nightmare….and I don’t think there is a simple answer to this.
I think Jon has the right attitude here – if there is a commercial server that you know of, or any evidence that hosting it in the UK would lead to improved performance then you’d be one ahead of the technical people we have employed so far. So that advice would be useful. But i afraid that we have exhausted the simple possibilities. For example IoPP simply will not hand out FTP access to an unknown number of people to its servers…or even to employees of the IoP.
Please let’s get back to Web 2.0 and collaborating on Physics, rather than spending time on Web 1.0 issues of downloads….which are a 1-off cost to most users.
A certain Mr Shirky thinks **that** is what we should be doing…
16 July 2010 at 15:24 #19148
I think this is an interesting and very on-topic discussion for physics. It is related to time, distance, signalling etc.. and a great example of physics in action. It is certainly on-topic for the thread and for the group, and Jon has said that he welcomes the ideas from everyone. It isn’t stopping me from discussing physics elsewhere on here or in any other medium like PTNC.
I don’t know who your technical people are, but it is common knowledge that hosting files close to your intended target audience improves performance. This is why the Amazon CloudFront service that is partnered with AmazonS3 file hosting exists. The trade off here is whether the improved performance is considered worth any additional cost (if it cost more) or slightly more hassle, if the facility isn’t quite as user friendly. Maybe that is the discussion that has gone on with your technical people?
16 July 2010 at 17:52 #19149
I was a bit intermperately tyring to fill out some of the background, to save covering the same ground over and over.
If it is common knowledge, then some evidence of the effect on 100 downloads of a 400m M file in a month would help the CBA. Physics needs numbers…so far the discussion has been long on assertion, short on evidence….
Mounting it on the same server is not an option.
Suggestions for UK based ftp substitute sites welcome.
16 July 2010 at 19:04 #19150
I think the element that was missing from my question earlier on today was the money bit, to be honest. The web has moved on so much since the 90s when I last did any significant ftp/piping/general command line stuff. You have answered my question nicely.
My other point about displaying the web address of the download site was more one of convenience than of security, I think. I was just thinking of a big round ‘click here to download’ button.
16 July 2010 at 20:16 #19151
The numbers for this argument are well known, by collecting more we would only be adding a spade-full of data to an existing mountain. Witness the high number of website mirrors so that you can download software from a server near to you, and websites that cleverly (and transparently) re-direct you to a server that is near to you by doing a reverse IP lookup.
That said, a UK solution might be outside of the budget. Hosting in the US is always cheaper. For what occassional benefit their might be, the financial cost may be too high.
However, as per my previous post. If someone tells me what features are required of a UK based service, I will happily look through the information I have collected over the years to see if there is a viable UK or European option.
19 July 2010 at 13:34 #19152
> numbers for this argument are well known
Can anyone point me at a simple document that reports download times for US vs UK, or the advantages of mirroring, or something similar?
The sorts of features we’re after are an easy Web-based user interface, allowing us to nominate collaborators, structure the filespace easily, tag the files, etc. The features available to us (not that we use all of them) are listed here.
19 July 2010 at 22:02 #19153
There is a Wikipedia article that explains the advantages of mirroring (or hosting in the country you are targetting):
Amazon CloudFront is described here:
Data on UK v US download times isn’t going to be easily available because it is data that changes over time on the scale of months, weeks, days and time of day. However, it is logical that the fewer servers and Internet providers that are in the chain to the file you are downloading the more reliable and faster the download will be.
You can test this out at busy times (try 7pm in the evening) using this website:
It will show you a map of the UK, but you can drag the map and test your download speed from servers all over the world. I find my ping time is 9 times greater to San Fransisco, and download usually 20-30% slower. Even ping times can make a difference to download times because of the return time for individual data packets.
You can also see a discussion here about the issues of ping times and server locations:
Having said all that, to get all the features you want at the price you want to pay might mean you are stuck with the US hosting of files anyway. However, I hope the discussion has been interesting
You must be logged in to reply to this topic.