Nixers project: Bittorrent library - Community & Forums Related Discussions

Users browsing this thread: 10 Guest(s)
z3bra
Grey Hair Nixers
Thanks for having think about the implementation!

I didn't think a lot about it myself, but my first idea was to go with your latest proposal: write directly to the end file, and allocate the full size (which is known) first. This has the advantage of failling quickly in case you don't have enough space.

First of all, let's use the correct terms. A "torrent" is the final file. Each torrent is splitted into "pieces", each having a hash, and each piece is sent as multiple "block" (see the RFC for details).

Your idea is to put each piece into its own file, and then, concatenate them when you jave all the parts. The main advantage I see here is that you can easily know which part you've already downloaded, allowing easy stop and starts without a "cache file".
The main disadvantage is that you will have a lot of I/O upon concatenation, and that would be a single threaded process.
This would also use a shitons on inodes, but that's not the biggest issue.

The "sort as they come" idea seems slow, bloated and complex to implement for no practical reason, so I'll just ignore it.

By writing directly to the file, you save some time, as when you'll receive the last piece, your end file will be ready. This is also the simplest approach for me, as you will know the size of pieces, and their position, so writing to the file is just an fseek() away. The problem is that this is not atomic, and require a "cache file" to remember which pieces are retrieved (altough you should be able to reconstruct it by checking which pieces of the end file are zeroed).

For now, I find the later simpler, as it eliminates intermediary parts from the process. It comes down to
receive a piece, check its hash, write it to the file, and that's it. Your idea adds another step which is "put all the parts together", and another step means more code, and thus more attack surface for bugs :)


I think there is no urge to settle this now.
For now, the structs for pieces/blocks are not yet finished. We also cannot talk to trackers, or other clients. The advantage of this RFC is that (I think) all parts are explained in a logical order. For example, you cannot understand how a metainfo file is written if you don't what bencoding is.
You're talking here about the way we write pieces received to the torrent, which comes, I think at 6.3.10 in the RFC ("Piece" message).
The library is currently stuck at point 3: "pieces and blocks". We have many things to do before deciding how to write them!

Implementing the full protocol is a big task, and we need to stay focused on what needs to be done if we want to get it working quickly.

I chose to implement all the points in the RFC one by one, in the order they appear. It might not be the best way to handle this project, but at least I can easily know where I'm at, and what needs to be done.
If you think of a better way, we can still discuss it though, I wanted this project to be the one of the community, not just mine.


Messages In This Thread
Nixers project: Bittorrent library - by z3bra - 30-07-2017, 04:40 AM
RE: New project: Bittorrent library - by r4ndom - 30-07-2017, 05:33 AM
RE: New project: Bittorrent library - by Tmplt - 30-07-2017, 07:20 AM
RE: New project: Bittorrent library - by z3bra - 30-07-2017, 07:22 AM
RE: New project: Bittorrent library - by Tmplt - 30-07-2017, 07:31 AM
RE: New project: Bittorrent library - by z3bra - 30-07-2017, 09:15 AM
RE: New project: Bittorrent library - by Tmplt - 30-07-2017, 09:45 AM
RE: New project: Bittorrent library - by z3bra - 30-07-2017, 10:42 AM
RE: New project: Bittorrent library - by z3bra - 30-07-2017, 08:45 PM
RE: New project: Bittorrent library - by venam - 31-07-2017, 12:22 AM
RE: New project: Bittorrent library - by z3bra - 31-07-2017, 03:36 AM
RE: New project: Bittorrent library - by jkl - 31-07-2017, 04:13 AM
RE: New project: Bittorrent library - by z3bra - 31-07-2017, 07:18 AM
RE: New project: Bittorrent library - by sff - 31-07-2017, 01:28 PM
RE: New project: Bittorrent library - by z3bra - 01-08-2017, 04:19 AM
RE: New project: Bittorrent library - by r4ndom - 01-08-2017, 05:31 PM
RE: New project: Bittorrent library - by z3bra - 01-08-2017, 06:19 PM
RE: New project: Bittorrent library - by z3bra - 02-08-2017, 03:53 AM
RE: New project: Bittorrent library - by r4ndom - 02-08-2017, 06:24 AM
RE: New project: Bittorrent library - by z3bra - 02-08-2017, 08:27 AM
RE: New project: Bittorrent library - by xero - 02-08-2017, 07:27 PM
RE: New project: Bittorrent library - by z3bra - 03-08-2017, 03:31 AM
RE: New project: Bittorrent library - by r4ndom - 03-08-2017, 04:57 AM
RE: New project: Bittorrent library - by z3bra - 03-08-2017, 07:59 AM
RE: New project: Bittorrent library - by josuah - 05-08-2017, 04:06 PM
RE: New project: Bittorrent library - by z3bra - 05-08-2017, 06:18 PM
RE: New project: Bittorrent library - by josuah - 05-08-2017, 07:21 PM
RE: New project: Bittorrent library - by z3bra - 06-08-2017, 07:10 AM
RE: New project: Bittorrent library - by josuah - 06-08-2017, 01:10 PM
RE: New project: Bittorrent library - by josuah - 06-08-2017, 05:00 PM
RE: New project: Bittorrent library - by z3bra - 06-08-2017, 06:40 PM
RE: New project: Bittorrent library - by josuah - 07-08-2017, 09:38 AM
RE: New project: Bittorrent library - by z3bra - 08-08-2017, 05:49 PM
RE: New project: Bittorrent library - by josuah - 11-08-2017, 05:23 PM
RE: New project: Bittorrent library - by z3bra - 12-08-2017, 03:37 AM
RE: New project: Bittorrent library - by josuah - 12-08-2017, 07:24 AM
RE: New project: Bittorrent library - by z3bra - 12-08-2017, 08:42 AM
RE: New project: Bittorrent library - by josuah - 12-08-2017, 07:06 PM
RE: New project: Bittorrent library - by nas - 24-08-2017, 11:53 AM
RE: New project: Bittorrent library - by josuah - 24-08-2017, 01:36 PM
RE: New project: Bittorrent library - by josuah - 24-08-2017, 01:42 PM
RE: New project: Bittorrent library - by nas - 25-08-2017, 02:44 AM
RE: New project: Bittorrent library - by z3bra - 25-08-2017, 10:18 AM
RE: New project: Bittorrent library - by z3bra - 26-08-2017, 03:26 AM
RE: New project: Bittorrent library - by josuah - 29-08-2017, 02:51 AM
RE: New project: Bittorrent library - by z3bra - 29-08-2017, 03:17 AM
RE: New project: Bittorrent library - by josuah - 29-08-2017, 08:16 AM
RE: New project: Bittorrent library - by z3bra - 28-09-2017, 11:53 AM
RE: New project: Bittorrent library - by z3bra - 09-10-2017, 02:36 PM
RE: New project: Bittorrent library - by josuah - 10-10-2017, 05:24 PM
RE: New project: Bittorrent library - by z3bra - 10-10-2017, 07:36 PM
RE: New project: Bittorrent library - by josuah - 28-03-2018, 05:40 PM
RE: New project: Bittorrent library - by z3bra - 03-07-2018, 11:24 AM