Nixers project: Bittorrent library - Community & Forums Related Discussions
josuah
Here are requests sent by bittorrent trackers:

rtorrent: https://p.iotek.org/fe2
Code:
GET /?info_hash=OG%15%00%E3%8E%88%BDd%03%AE%3A%E4%CBz%9Al%BDsv&peer_id=-lt0D60-%3C%FE%A06%94-%CC%9Bi%DC%0A%8C&key=6cde84e1&compact=1&port=6915&uploaded=0&downloaded=0&left=3825205248&event=started HTTP/1.1
Host: 0.0.0.0:1234
User-Agent: rtorrent/0.9.6/0.13.6
Accept: */*
Accept-Encoding: deflate, gzip

transmission: https://p.iotek.org/bd6
Code:
GET /?info_hash=OG%15%00%e3%8e%88%bdd%03%ae%3a%e4%cbz%9al%bdsv&peer_id=-TR2920-iesno8z0fsbc&port=51413&uploaded=0&downloaded=0&left=3825205248&numwant=80&key=52541b14&compact=1&supportcrypto=1&event=started&ipv6=2001%3A4b98%3Abeef%3Aa%3Aca5b%3A76ff%3Afe48%3A2a0d HTTP/1.1
Host: 0.0.0.0:1234
User-Agent: Transmission/2.92
Accept: */*
Accept-Encoding: gzip;q=1.0, deflate, identity

There is one empty line at the end of each request, and lines are ended with '\r\n' as wanted by HTTP.

[EDIT] it seems that z3bra already implemented this part :]
nas
Thanks for the info @josuah. I don't have anything to do at work but our network blocks many addresses and ports. I'll just read more for now.


I can't even connect to the IRC servers. gahd.
z3bra
The GET requests sent to trackers is indeed working since commit 913fcb4. It should now support any kind of message, and it returns the next interval to wait before the next heartbeat request. There is currently no way to retrieve other parameters like the number of seeders/leechers though.

I'm currently working on PWP messages (handshake seems working now)
z3bra
I'm currently facing some decisions, about how to use the library. These decisions will define how the PWP functions will be used. Here is a pseudo code of how I imagine the interface. Feel free to discuss:

Code:
int interval = 0, last = 0;
uint8_t msg[MESSAGE_MAX];
off_t offset = 0;

interval = thpsend(torrent, "started");
last = time(NULL);

/* everything is wrapped within an infinite loop
for (;;) {
    if (time(NULL) - interval > last)
        interval = thpsend(torrent, NULL);

    foreach(torrent->peers as peer) {
        if (!peer->connected) {
            pwpsend(torrent, peer, PWP_HANDSHAKE);
            pwpsend(torrent, peer, PWP_BITFIELD);
            pwpsend(torrent, peer, PWP_INTERESTED);
            pwpsend(torrent, peer, PWP_UNCHOKED);
        } else {
            if (!peer->ischoked)
                pwpsend(torrent, peer, PWP_REQUEST); /* Request a new random block */
        }

        switch (pwprecv(torrent, peer, &msg)) {
        case PWP_REQUEST:
            pwpsendblock(torrent, peer, msg);
            break;
        case PWP_PIECE:
             offset = pwprecvblock(torrent, peer, msg);
             if (sha1(torrent->pieces[offset]) == torrent->sha1[offset]) {
                 endgame(torrent, offset); /* sends PWP_CANCEL message to all peers */
                 interval = thpsend(torrent, torrent, "completed");
            }
            break;
        }
    }
}

Does that make any sense? Would you change anything?
josuah
This raises a few questions to me: what if the client's or the peers network is slow or a peer does not respond?

If pwpsend calls are blocking, the client will hang and wait for the response of every client before continuing with the next request, while it could send them all and wait for all answers, while still proceeding to the downloads (which would ideally never stop).

But the primitives looks good, it may be possible to use them to build asynchronous clients. I'm not yet very experienced with fork() and shared memory, but this is how I picture it based off what you propose:

Code:
/* setup shared memory: for the count of parallel requests, blocks requested... */
for (;;) {
    foreach(torrent->peers as peer) {
        if (count < MAX_REQUESTS && !fork()) {
            count++;
            if (!peer->connected) {
                pwpsend(torrent, peer, PWP_HANDSHAKE);
                pwpsend(torrent, peer, PWP_BITFIELD);
                pwpsend(torrent, peer, PWP_INTERESTED);
                pwpsend(torrent, peer, PWP_UNCHOKED);
            } else {
                if (!peer->ischoked)
                    pwpsend(torrent, peer, PWP_REQUEST); /* Request a new random block */
            }
            count--;
    }
}

And the same style for the other calls.

Another solution would be to have "pwpsend()" and friends to be asynchronous thmself, but if PWP_HANDSHAKE, PWP_BITFIELD, PWP_INTERESTED, PWP_UNCHOKED needs to be sent in this order, you would need shared with a queue for each peer listing all operations to send in turn, with a limit for not making the queue too long (requesting a piece 38 586 times to a peer not responding and still stuck at PWP_HANDSHAKE, ...).

In either case, this looks a really sane and simple interface, without complex structs to setup, neither initlibgbt() so far.

I am not good yet at networking or async programming, so there may be better/more standard ways to achieve it.
z3bra
pwpsend() call are not not blocking. They only send a message and don't expect any response. pwprecv() and friends are though. It would indeed be a better idea to make them non blocking using poll()/select() (that would be up to the user implementation then).

About the network request queue, I though about it as well. Some recommendation advise no more than 10 request queued to avoid complex overhead. Not sure how to implement this properly though.

I'm definitely not for adding fork()/pthread_create() calls IN the library. This should be the user's choice to do so. But we could indeed design the library around it, using pwppreparereq() + pwpsendreq(), and let the user decide when he wants to send the messages.
josuah
Good point: keeping more out of the library. Then the library and client complexity do not to stack together. No "do the stuff" function that manages everything. :)

The queue can also be managed by the client. As an example, you might want to use the lib to try to overflow another implementation to test the request load it can handle.

Are the request also non-blocking if the client is not connected to internet anymore (and unable to receive packet at all)?
z3bra
Some update on this project!

Libgbt can now download, and upload torrents! It only handle it when there is a single peer though, we need a way to prevent querying the same piece/block to multiple peers.
I didn't test multi file torrents in this configuration, but it should technically work :D

The lib also doesn't bind on a port to accept incoming connections. I will only connect to peers, but won't accept new connections (not useful for seeding)
z3bra
We reached a milestone here! libgbt can up/download torrents, with multiple peers. It can accept new connections as well (when seeding only for now).

Basic functionality being there, let's fuck it up and rewrite some bits! Here is what I'm not happy with, or what I want to add/improve:
  • out request management (ring buffer?)
  • peer list management
  • keep track of requests
  • end game algorithm
  • in/out net throttling
  • rework grizzly API

I'm not sure how the API should be used. I'd preffer something as simple as possible, like the following:

Code:
struct torrent to;
grizzly_load(&to, "file.torrent");

/* start download */
while(!grizzly_finished(&to))
    grizzly_leech(&to);

/* torrent is complete */
for (;;)
    grizzly_seed(&to);

If anyone has an idea or input, I'll welcome them!
josuah
"I have long been of the opinion that a good software project must be written at least three times: once to understand the problem, once to understand the solution, and once to actually write it properly."

https://bearssl.org/goals.html




Members  |  Stats  |  Night Mode