Solidity in 2021

Transcript

foreign foreign welcome to my talk my name is Kristen reichwisner I joined the ethereum project in 2014 and created the solidity programming language and in this talk I would like to tell you yeah essentially what you were up to in the past few months and what are the plans for the near future the solidity team currently consists of 12 core members but of course we have many contributions from the open source community and we're a very open team so

if you have any I don't know questions remarks proposals or whatever please don't hesitate to reach out to us there are various channels um yeah you can either use Twitter or our Matrix slash guitar chat room and we'll set up a forum recently sorry and uh yeah some time ago we noticed that there is a lot of scattered information about solidity and that's why we created the solidity portal at soliditylang.org it now contains all the yeah links you need to have um

it combines our our blog our Forum the chat room the documentation uh the binaries can also be found under that domain and the website even contains a small yeah text editor uh with an integrated compiler where you can try out small Snippets of solidity um the so it was not like that in the beginning um in the beginning I just yeah did some things and talked to the I don't know 20 to 50 people in the ethereum community about new features of solidity and what they

like but this is getting more and more difficult um first of all to find a the people to talk to and also to yeah we have to be much more carefully not to break things and so already I think maybe two years ago three years ago we started the mailing list to include some yeah users who want to give feedback about the compiler about the language this did not turn out so well so the list still exists but it's kind of dead and next we tried a forum uh this works much better I would say so uh it's at

forum.solidylang.org and um so this is not really meant as a kind of support Forum but uh still if you have uh essentially if you have questions that can best be answered by the compiler Engineers themselves then please go there and don't hesitate to open a new topic or something like that but we would also like to use this forum or we also do use this forum to propose new language changes um and uh try to get feedback from the users there so if you want to participate in the Forum please sign up there then the next thing we tried is uh the solidity Summit which was planned to be

a in-person event uh with workshops and talks and whatever in Berlin in April 2020 um of course this did not turn out uh like that so we turned it into an online event and I think it was a big success so you can the the videos so we recorded everything the videos are still online on solidity lang.org um yeah then another thing we tried is uh kind of uh first get an impression of how the solidity Community looks like so we did this user survey

um which yeah contained questions like uh how long have you been programming in solidity what is your general programming background which resources do you use and so on which country you're from and this uh yeah provided this very provided us with very useful data about the the community in general so kind of as a a baseline to to build on top of um we also created individual small Channels with different focused group like Auditors uh solidity tooling Yule users which is also very nice because um you could do this in the main

solidity compiler development discussion Channel but of course there's a lot of noise and those people don't really want to read through all these messages and then the most recent thing we tried is um so we realized that uh when you just ask questions to the general public hey how do you like this feature then you usually don't get really a response or perhaps if you have questions like hey um we have this feature but there's this problem do you know any solution to that then nobody answers of course so uh we are selecting uh certain individuals and we're having one-on-one feedback

sessions with uh yeah different people and there of course they can't escape so if you ask a question to a person directly then that person has to answer um and we're getting very valuable feedback uh new proposals for features so this uh is a really good thing um if you want to be on such a one-on-one feedback session then please don't hesitate to contact us okay um one yeah one question that came up a lot from our side in these feedback sessions is the question about why are you using such an old version of solidity uh it is

a complex topic we are aware of the problems we're trying to solve them um but on the other hand we also noticed that uh people were using a yeah more or less recent version of solidity but still they didn't know about all the features that were present in this version so um the next section of my talk serves two purposes first uh tell you features that you could already use but but don't know about and second give you a glimpse on what features you could use if you would just upgrade the compiler version um yeah the first one is of course checked arithmetics present since solidity zero eight um I hope everyone knows about that so

since solidity zero eight all arithmetic operations on Integer are checked against overflow and if an overflow happens a runtime error is triggered which reverts the current transaction or message call and it is important to know that the Overflow check is not performed on the on the evm word so not on the 256 bit word but on the actual type of the variable so in this example here if we have a 16-bit variable X which has the largest possible value for this variable and you add one then this will trigger a yeah a panic so this is something so ipanic is a new type of error that we introduced

which I will also yeah which you will see uh later in the talk um and you can so yeah and also important to know so the Overflow check is performed on the type of the operation and the type of the operation is always either the type of the left or of the right operand there are exceptions for exponentiation but let's not get into that and um yeah so okay yeah so it's not I don't know performed on if if this would be you and uh 64 y equals something then it's still performed on the un16.

and you can so these checks are of course more costly than not doing them uh so if you have uh I don't know code that does a lot of computations and you don't care about overflows or actually you want the wrapping Behavior this might be relevant in cryptographic code then you can wrap the code inside an unchecked block and disable the Overflow checks and yeah move to the old wrapping behavior of arithmetics um the current code can generate redundant checks so I don't know you due to a previous check you might know that the value in a variable is not very large but the compiler will still

generate the Overflow check for this variable uh we hope that this can be eliminated with the upcoming UL compiler pipeline and then uh yeah this will be on many of the slides feedback from you the question is uh whether in addition to overflow checks we also want to do value range checks on explicit type conversions this has been requested by some some other say yeah that's not really needed if you have any feedback on that please talk to us another very nice feature is the the immutable keyboard this has been in the compiler for quite some time it is both

a safety and efficiency feature because it um so you use it on a state variable and it will enforce that this state variable can only be written Once In The Constructor and later on it can only be read and this property is not enforced by the compiler it's actually enforced by the evm so it's a very very safe property and on the and the second bonus is that it's also extremely cheap to use So reading from an immutable uh variable costs three gas as opposed to 2100 gas if it would not be an immutable variable um yeah again feedback

uh there have been requests to allow So currently these variables can only be written Once In The Constructor and they cannot be read at creation time the reason is that we want to avoid problems where you read from the variable before it has been assigned um is this something so would you like to read from YouTube overalls at construction time or not we would still like to prevent reading from it uh before it has been assigned but this is yeah this is a property that has to be checked in the compiler and is a bit tricky so we don't want to prioritize this too high if you don't want to have it um yeah and then another nice feature is

user defined errors um and this is a new way to transfer or yeah report information back to the user about why a transaction failed um the most simple way to make a transaction revert is uh the require statement here in its conditional form so you require msg.weli to be at least price and if not then the the call Will revert and in this case it will uh send nothing back to the caller and I would rate this on the information scale as one out of 10.

and then there's this other way where you can provide a string literal together or a string together with the require statement um here the user will see oh yeah uh this reverted because I need to provide more ether um I would not I would still not call that information counter 10 out of 10 you will see later why and uh yeah the this string will be encoded in a similar way as a call to a function that is yeah called error and takes a string parameter and we introduced that in order to prepare for uh the thing that will now be on the

following side and this this is user-defined errors and uh so with user-defined errors you can Define your own error types and use the the error keyword for that and then you it is is actually very similar to uh declaring events or anti-clairing and using events so here we Define the error too cheap and of course as with almost all elements in solidity you can use netspec annotations to to document uh what is actually going on um and you use the custom error using the new revert statement so you just write revert to cheap and then

parentheses um I would say this conveys so because you provide the net spec string this conveys exactly the same information as the the previous uh or at least maybe even a bit more as the previous uh way to do it with the big difference that the net spec string is not stored in code so it's much much cheaper and you can actually you can make the net spec string as long as you want uh at no additional cost and uh yeah but this is not the end of it so it's even uh it gets even better and the reason is you can add parameters to custom errors so here we have an error too cheap and

it has two parameters provided and needed so uh essentially yeah you provided this amount of ether but actually needed or required is a higher amount and the cool thing is that um these you can provide values to these parameters that are runtime values so you you the user will see um I did not send enough ether and this is exactly the amount I sent and this is exactly the amount that was required so I only sent two ether but the contract actually needed four and uh if four is a constant then this is not a big gain but I don't know if it depends on some exchange rates or whatever you see exactly uh the yeah the value that

was required here and exactly where the problem is and I hope that um block explorers and other tools will get to a stage where we see these netspec error messages rendered nicely so and not only display these netspec messages but also um evaluate the template so here we have provided provided in backticks but needs needed in backticks ether and the block Explorer can actually extract the these runtime values MSD value and price from the revert data and substitute them into the string so you see the actual numbers uh that are there

instead of the the yeah constant string now um what are the cost of these operations um I would like to use the so I did some guess measurements maybe your gas measurements are different this of course also depends on Optimizer settings and other things but a I would like to yeah if you look at the costs as additional costs on top of the most simple require statement then um you see that the yeah that uh the custom error with parameters is both shorter and cheaper so the size is in bytes and the cost is in gas it is

both shorter and cheaper than the required with string and uh yeah but you also get as I said the the actual runtime values that cause the errors okay and then also something that is maybe not that widely known is that over the past versions we allowed more and more um elements of the language to be defined at the level of a file instead of inside contracts only since zero six you can already already use structs at the file level since zero seven one you can use functions at the file level and since zero seven four you can use constants at the file level and

uh I I personally particularly like the functions at file level um the way they are implemented is very similar to how internal Library functions are implemented um and the cool thing about them is that since they are outside of a contract you immediately see that it it cannot have access to the contract itself so because of its scoping because it's scoped outside of the contract there's no way it can access the state variables of the contract it can still access storage if you pass it an explicit storage pointer like in this example and uh so there's there's maybe one downside of these functions at five

level and this is also again something where we need your input um currently functions at file level are allowed to perform external calls to other contracts and of course they have to do this external call in the context of another contract and this is the yeah contract the function is currently executing in so um the kind of danger of using functions at file level is that these functions can assume the identity of the contract they are compiled into not sure if this is a real danger um but yeah as I said uh if you have an opinion on that then please contact us we also plan to extend the using statement so uh the using statement allows you to attach functions to types so instead of

um name of of P you can write P dot name of this is yeah maybe like like in Python where you have this explicit self variable and currently it's only allowed for Library functions or only for libraries I would like to extend this for uh yeah functions at file level or even all functions in a module now uh you might ask that uh if you declare all these stuff at file level you kind of lose scoping and the risk of name collisions is uh rather high but um the thing is that this risk only appears if you use the import statement like in this first

example here so if you just import a file without anything else this is the the simplest import statement but I think yeah many other languages do not recommend such Imports python doesn't recommend that JavaScript doesn't recommend it and um instead you can import only specific names from a file or you can also import another file as a module itself so in the third example here if you use import participant.sol as participant then you can access the elements of this file as participant. participant or participant or name of

we also plan an a an export statement which yeah I don't know the current plan is that it can be used once in a file and it has to list all the names that are imported by default if you just import the file Some people prefer the export to be named multiple times in a file again would be happy about your feedback now uh and so the yeah this slide is about a rather recent feature so these are I think these are yeah and the first one not but the others are zero eight I think four

features um we have support for call data slices so these are slices uh very similar to how they are in Python where you can reference a part of a byte array we have not implemented them from memory yet so only for call data um because it's a bit more complicated in memory then you can now convert byte arrays so the bytes type to a fixed byte array bytes four for example and this will always truncate so uh this these bytes four of msg.data will retrieve the first four bytes in the uh in the call data and then uh the third feature we we realized that so there's this function

ABI dot encode pact which does the the weird non-standard or I don't know packed encoding and um we kind of want to get rid of it and we we checked how it is used and we realized that the main thing it is used for is to concatenate by the race and then we thought yeah okay maybe just let's introduce introduce a new function for that that is called bytes.comcat so um bite some concat Works more or less that a more or less the same way as ABI dot encode pact but it only allows byte array-like objects like uh yeah byte Rays themselves or string literals or fixed byte types and I would like to

show you an example of how this can be used nicely yeah um this is unfortunately not the most recent version of the slides so um but let's take a look at it so this is a forewarning contract it takes a a Target address and a call data byte array and in the so if you take a look at the else case then this is the the simple case it's just forwards the call to the Target and but uh it has a special case where it checks if the function to be called is the transfer function of an erc20 contract and if that is the case then it

injects a different receiver for the transfer function so it it takes the the first four bytes of the payload which is the the function selector and then it adds the receiver and the rest of the payload and uh since the receiver is an address it cannot be used in bytes.comcat and so you have to convert it to a byte like objects first and this is best done with abi.in code and yeah I think previous versions or if you if you want to do the same thing with an earlier version of solidity it would have been yeah would it would look very weird and I think these features

combined provide a very nice way to uh yeah easily understand such contracts of battery monitor byte array manipulation okay um the solidity Yule IR pipeline we have uh talked about this a lot uh in the past months but I think now it's uh really almost finished um what am I talking about um the current solidity compiler takes solidity and directly transforms it into a stream of evm opcodes this was nice in the beginning because it yeah was a bit easier to write and it generates quite a good code but

nowadays you want a bit more flexibility and that's why we introduced an intermediate layer so an Intermediate Language where we first compiled solidity to Yule which is essentially inline assembly and then from Yule to evm bytecode and one of our main goals is that um this Yule code is still readable so uh the cool thing is you can actually audit assembly code that is the output of the of the contract and it also provides flexibility because you can you can first compile solidity to you and then modify the your code and then continue the compilation and this

modification could be either injecting logging information for a debugger or replacing storage modifying um uh function calls by by yeah something else that makes it usable on level 2 for example and um but we also think and also have a proof in the meantime that this allows for uh better optimization of the code because Yule does not have indirect jumps and this allows you to do cross-function optimization okay how far it is we have complete solidity language support since February

and every new feature we add has to also compile yield and it has not been so you can already use it but it's it's marked as experimental and uh the reason is that we're still working on uh providing all the debugging information uh that the old compiler pipeline has and we also want the compilation to be a bit more efficient so we're currently we're putting a lot of effort into um yeah finding a good transform from your code to evm code which does better stack slot allocation than the current uh compiler does and um yeah we're currently at a stage where at least for our uh test Corpus we have comparative or even

better gas costs for the new compiler pipeline than for the old if you want to use it then the compiler switch is called experimental wire ir and if you add this switch then you can use the the regular output selectors like bin or Asm and if you want to take a look at the the intermediate code itself you can use the minus minus IR switch or if you want the code after optimization you use the minus minus IR minus optimized switch and one important fact here is that if the optimizer is turned off then the non-optimized and the optimized code is the same so you also have to activate the optimizer using minus minus optimized if you want

to have the optimize code okay some upcoming features um we finally want to release a minimal viable version of fixed Point types the issue number is 409 if you have some opinions there please go there we want to get rid of the stack to deep error and the solution to that is the stack to memory mover one important thing is here uh the so this element moves stack variables and converts them to Memory variables the problem is it has to allocate uh a place in memory to store

this variable and if you use inline assembly without allocating memory this will Clash so this feature I mean it's currently disabled in general but we have to be very careful with inline assembly and if you want to be compatible with future versions then please try to also always allocate memory in an assembly I already talked to uh how to extend using four then one item that came out of one of these one-to-one feedback sessions are user-defined value types which are where you can define a type for for time or for uh currency amount or something like that is based on top of an existing

value type so you say a new type called time but it's an it's actually a uint as far as data representation is concerned and one question here is whether or not we also want operators on these newly defined types if you have an opinion on that please go to issue number 11531 um then templates And Trades uh have been an long-standing uh feature you would like to have um also data enums so these are also called Union types or yeah where you can we have an enum but you also can add struct like data to each enum value um

yes yeah and uh yeah if you want to reach us then either you directly go to the issues or our Matrix chat room or the Forum on our website then the last topic I want to talk to you about is uh the solidity optimizer people recommend to switch off the optimizer when compiling solidity because the optimizer can have bugs while this is true in principle I would like to talk to you about the uh yeah quality of this or the quantity of that

statement um I found an academic article I'm not sure how how exact these numbers are but it says that in the past four years the GCC compiler had 800 bucks in its Optimizer component and the llvm compiler had 180 bucks in its Optimizer component and now my question is to you is how many bugs have been found in the solidity Optimizer component in the past four years which means uh starting from and including the version zero four eleven just just Shout yeah zero I'm not I mean we're not

perfect yeah someone saw the slides right oh it's five bucks uh so this is a bit more than one Buck per year and I would actually even like to talk to you about the specific bugs okay no sorry um and uh also one important thing about these five bucks is that all of them were found by fuzzing by our internal team efforts for fast testing so none of these bugs were reported from outside and I'm also not aware of any of these bugs being present in in production smart contract code okay the the first one was uh or I mean

the most recent one was found in March 2021 and is a very notorious bug which called ketcha caching bug it is presented every single version of solidity up to zero A2 and the way to trigger is the way to trigger it is as follows you need a pair of catch actor 56 calls and they cannot be separate by any jump or any memory operation that is more complicated than a single than a simple simple M store and um they have to have they have to both have a constant length that is known by the compiler at compile time and um if they refer to a memory location where

the compiler knows that the content is the same then the optimizer Compares this content um and it rounds up the length to the next multiple or 32 in instead of taking the exact length into account and yeah if the if the optimizer thinks that the memory contents are the same then it replaces the second call by the the result of the first call okay uh then the the second one is you will optimize a redundant assignment Eliminator bug um this was a bug in the Redundant assignment elimination routine which could be triggered uh when they were

break or continue statements but I have to say that the you will Optimizer was flagged non-experimental two weeks before this bug was fixed and the optimizer was only so it was only activated for the ABI encoder and not for inline assembly and it was activated for Standalone duel mode but the ABI encoder never used any break or continue statement so the only way to trigger this was through Standalone Euro mode which which uh yeah was considered experimental then December 2019 there was an M load s load redundancy bug in dual Optimizer again related to you to to Loops at that

point the Yule Optimizer was still considered experimental but we still want to wanted to have this uh bug in the in the official list and in March 2019 um there was a bug in the yeah and it's match expression based simplifier where a byte opcode was incorrectly replaced and the byteop code extracts a single byte from an evm board and the problem was triggered when the byteocode was applied to a value of 31 so a value known at compile time and constant of 31 and not the index so not the the byte but the value the byte is

extracted from so I'm not sure how often you extract a byte from the value 31. and the last one uh was related to combining two shift left operations so you combine two shift left operations by adding the amount your shift by and the bug was that overflow was not taken into account in this Edition so if you shift left by 2 to the 256 uh bits and then shift left by another bit then the optimizer was would reduce this to a shift by by zero instead of uh yeah I shift by a number that cannot be represented in the in an evm word so yeah shifting by more than 32 bytes is not really useful uh sorry but yeah

or by more than 256 bits so um yeah but you have to shift not by 256 bits by two but instead by two to the 256 bits okay as a summary all of these five bugs combine the following properties first they were found by fuzzing and not in real world code so the question is whether anyone would write code that could trigger these bugs at all second they have very obscure conditions to trigger and uh third and I think this is the most important point if you have full test coverage on your smart contracts then you will notice that something weird is happening because the the optimizer Optimizer just replaces The Code by something completely

different um and also over the recent yeah over these four years we ramped up our fast testing and also the formal verification of the optimizer routine and I hope that we're currently catch all the bugs before the um uh before the version is released or uh even before the pull request is merged into develop branch and yeah so why am I talking so much about the optimizer in in my opinion the optimizer is not about saving gas so we didn't write the optimizer so that your contracts get cheaper we we did write the optimizers that your contracts get safer and the reason is if you have if

you have two uh versions of the same routine and one version is uh I don't know handwritten inline assembly that heavily optimizes for gas and the other version is and and this this handwritten version it has many comments to explain why it is correct and the other version is just plain solidity and it's obvious that it's correct and now if the optimizer is capable of combining these two versions into essentially the same evm byte code with the same gas cost then why would you not choose the version that is obviously correct and yeah I hope that um yeah this will motivate you to activate the optimizer to increase the safety of

your contracts instead of disabling it to increase the safety of a contracts okay this is the end of my talk I do not only want to thank you for your your uh patience uh for listening to my talk I also would like to thank the solidity team for constantly improving the compiler and the language and and I would also like to thank everyone who interacts with us to give give feedback and contributes in improving the language and yeah last but not least I want to thank the the whole awesome ethereum community so thank you for the talk it's not so

much a question as a comment about the language and what you presented uh there have been very frequent release of measure versions of solidity over the past year we are now at 8.6 so hopefully eight zero eight we'll have a longer lifespan than zero seven uh still you discuss the way that you want to remove ABI dot encode pact for bytes concat as far as I understand those are very very similar however removing abi.10 code packed remove backwards compatibility so it's a breaking change requiring people to update their version it's always good to be on the last version but having to update also decreases ability for user to compose their code with other projects that use

a previous version of solidity so I want to understand better why it's needed to remove it and have your opinion about the effect of removing such features on on compatibility for for Developers yeah thank you this is exactly the the kind of feedback we need um we are aware that this creates problems at the same time we want the language to evolve um I mean we can so the usual way to remove something is of course to first introduce it with a warning and then remove it in the next next breaking release um but if the community says yeah let's just leave it into a half backwards compatibility uh we can also do that so I think this all this this upgrading

thing uh I think we need to sit together with all the involved people and really find out what the problem is I mean also one problem is this this insane long audit cycle where people have to wait uh half a year to a year for an audit and can't upgrade the solidity version I think this is also we some somehow have to solve as a community and this plays into this problem foreign thank you