Please login or register.

XMR Pool Infrastructure Overhaul

Update 2018/6/12: this FFS was paid to hyc nVidia solo miner project

This is a proposal that I'm going to do in parts, due to the nature of it - I won't know how much work there is to do, or what milestones to set, until the first step is done. So, first, let me describe the problem XMR has.

All PoW mined cryptocurrencies today face the possibility of 50%+ attacks. While they're called "50%+ attacks," or "51% attacks" - this is a misnomer. Having greater than 50% only ENSURES your attack will succeed. Success can be likely with even 30% of the network hashrate, especially if reversing few confirmations. The attacker has infinite tries, and only needs to get lucky once. Now, a single entity amassing this much hashpower becomes more and more unlikely as the currency grows, but we face more of a threat from the pools the miners use.

XMR currently has what is referred to as "stratum," but is nothing of the sort. It is effectively Getwork over TCP. Now, the scalability issues aren't much of an issue here as they were with Bitcoin - as XMR miners tend to mine in KH/s at the most, miners will hardly run out of nonces to try. But the greater threat here is that the miner knows NOTHING about what they're hashing. They will blindly solve hashes for the pool, even if the block being mined is for attacking the network. Additionally, block creation is not decentralized in any way - the pool chooses the transactions which go into the blocks, as well - this is a disaster for decentralization. On top of all of this, the current mining infrastructure code is so badly put together that it will break in the future - exactly when, I've not calculated, but XMR's block header size, and the location of the nonce, is not constant. When the timestamp grows by a byte, and it will, most miners will cease to mine valid blocks.

Clearly something needs to be done, but what? I propose a heavy modification of Getblocktemplate (GBT) from Bitcoin. The only thing that can be taken from it is the basic ideas, because of how different XMR and BTC are, but the goals are such that miners who run daemons/wallets will check the pool's work against the network, as well as create their very own custom blocks. The pool will still be accepting shares, the same as before, but instead of checking with a block header they sent you, they will check your block proposal and ensure it pays the pool. This means miners may include their own transactions, their own coinbase signatures, and examine every detail of the pool's work that it's handing out. If even a small portion of miners run daemons/wallets, allowing the miner to police the pool, then they can warn everyone else if a pool becomes malicious. Since attacks take time and sustained hashrate, this guts the effectiveness of abusing innocent miners' hash for a malicious attack.

Now, in order to determine how to proceed, I need to do a LOT of research. I need to know the exact layout of blocks, transactions, and how blocks are built from the ground up. This is because block creation will have to be implemented in the miner - I'll also need to consider pool side verification of shares, and the best way to implement communication between the miner and pools. I'll have to bring that all together to create a technical specification for the new protocol - for everyone to be able to implement. After this is complete, I probably will post another thread proposing the relevant development.

So, is this something the community would like to pursue?

EDIT:

As a footnote, here's a comment I posted with proof of the mining code breaking:

For you to look, here's the relevant details:

Here's where the miner hardcodes that the nonce is 39 bytes into the block header: https://github.com/lucasjones/cpuminer-multi/blob/master/cpu-miner.c#L1071

And here's the block header format, showing that before the nonce, there are three variable-sized integers: https://github.com/monero-project/bitmonero/blob/master/src/cryptonote_core/cryptonote_basic.h#L289

Really, all the current infrastructure has had corners cut at every possible opportunity.


I propose 750 XMR in total over two milestones here. The first milestone will be supporting documentation, and will help XMR in more than one way: more of the data structures and the internal workings of Monero will be documented for people wishing to study it for other purposes than mining. This will include the format of blocks, which requires the format of Monero's variable-sized integers to be documented as well. Also needed for this milestone is documentation on difficulty (difficulty to target and back, calculating average network hashrate from difficulty, this sort of thing), and detail on creating AT LEAST the coinbase transaction (the one that pays the miner.) Full documentation on transactions is not necessary, but would be nice. These entries are to be posted on monero.wikia.com.

The second milestone I propose is the details of the protocol between the pool and the miner. This will include the format of the information and what sorts of information need to be passed between the miner and pool. It will also detail rules for communication between them, and detail how to construct block proposals for the pool, etc.

450 XMR for the first milestone, as it requires the most research and time put in, 300 XMR for the second.

Replies: 29
Gingeropolous edited 8 years ago Weight: 0 | Link [ - ]

WARNING

The lack of activity on this project has put these funds in danger of being re-appropriated.

If any funders significantly disagree, please voice your concern. Funds will probably be moved to pay for hycs work on the solo nvidia miner (which, imo, is well deserved and sort of also gives a nod to the fact that he's, in general, doing awesome monero developing and stuff)

This comment has been stickied at the top of the thread! Go to post

nagrom1981 posted 8 years ago Weight: -61 | Link [ - ]

Donated.

Drhiggins posted 8 years ago Weight: -62 | Link [ - ]

Donation sent.

Dufkin posted 8 years ago Replies: 1 | Weight: -111 | Link [ - ]

Thank you Wolf0! I'll support this!

Reply to: Dufkin
Dufkin edited 8 years ago Weight: -65 | Link [ - ]

glad to see this is going live. sent some xmr as promised :)

rocco posted 8 years ago Weight: -111 | Link [ - ]

thank you wolf0 for working with monero. most of us who are here for some time have respect for your work and know you are a real expert in this.

thats why i support your effort. mining infrastructure has to increase, we should treat this as a security issue.

supporting decentralization where possible is something we always have to do!

Wolf0 edited 8 years ago Replies: 1 | Weight: -117 | Link [ - ]

Fun fact: In my poking around for the solo miner, I discovered that the pools all show a slightly wrong nethash due to dividing by 1024 instead of 1000. Threw me off for a bit until I realized it was wrong by ~0.5MH/s.

Reply to: Wolf0
Drhiggins posted 8 years ago Replies: 1 | Weight: -117 | Link [ - ]

I've always wondered about this because my reported hash rate on the mining pools is always lower than my actual hash I see on screen. Good info to know.

Reply to: Drhiggins Wolf0
Wolf0 posted 8 years ago Weight: -114 | Link [ - ]

I believe that's actually because it's estimated based on shares, but they might fuck the divide there, too. Not sure.

Drhiggins posted 8 years ago Weight: -117 | Link [ - ]

If this was set as a bounty/project for funding I'd donate. Since half the XMR I hold have been mined I'd like to make the network as healthy as possible.

dEBRUYNE posted 8 years ago Weight: -119 | Link [ - ]

Supporting this as well.

Hueristic edited 8 years ago Weight: -119 | Link [ - ]

How imminent is this?

BTW: Wolf0 thx for pointing this out.

luuul posted 8 years ago Weight: -119 | Link [ - ]

I also support this!

dEBRUYNE posted 8 years ago Weight: -119 | Link [ - ]

Some clarity from IRC, see bitcointalk link -> https://bitcointalk.org/index.php?topic=583449.msg12850626#msg12850626

For some reason I am borking the forum if I just copy paste it.

theBunk posted 8 years ago Weight: -120 | Link [ - ]

Hi.

"When the timestamp grows by a byte, and it will, most miners will cease to mine valid blocks." <-- this sounds serious.

As a community member, i believe pursuing this would be a strong course of action.

kind regards,

Gingeropolous posted 8 years ago Replies: 1 | Weight: -120 | Link [ - ]

As I've stated elsewhere, I think the mining infrastructure of monero is currently underdeveloped, and the less effort put it into it now, the more likely it is to run away and turn bitcoin-like or to break entirely (as pointed out by wolf above). Thus, I think its something that should be pursued.

Reply to: Gingeropolous
Wolf0 posted 8 years ago Weight: -120 | Link [ - ]

For you to look, here's the relevant details:

Here's where the miner hardcodes that the nonce is 39 bytes into the block header: https://github.com/lucasjones/cpuminer-multi/blob/master/cpu-miner.c#L1071

And here's the block header format, showing that before the nonce, there are three variable-sized integers: https://github.com/monero-project/bitmonero/blob/master/src/cryptonote_core/cryptonote_basic.h#L289

Really, all the current infrastructure has had corners cut at every possible opportunity.

pa posted 8 years ago Weight: -120 | Link [ - ]

Sounds worth doing. Thanks for writing this up.