channel logs for 2004 - 2010 are archived at http://tunes.org/~nef/logs/old/ ·· can't be searched
#osdev2 = #osdev @ Libera from 23may2021 to present
#osdev @ OPN/FreeNode from 3apr2001 to 23may2021
all other channels are on OPN/FreeNode from 2004 to present
http://bespin.org/~qz/search/?view=1&c=osdev&y=18&m=9&d=20
12:05:07 <geist> well, there ya go
12:06:11 <graphitemaster> zid, the fqa is highly regarded as one of the worst resources on C++ in the C++ community. Go join the C++ Slack and ask about it, I doubt you'll last long before you're banned
12:06:45 <booyah> or just join ##c++ like a normal person
12:06:51 <graphitemaster> yeah that works too haha
12:06:52 <booyah> not a Slack hipster numale
12:07:28 <zid> "Join this place full of stockholm syndrome victims, and ask them about counter insurgency, they'll hate you"
12:07:28 <zid> no shit
12:10:31 <graphitemaster> "join this place full of people who know the language and ask them about why this thing is wrong, nah forget that I'll just chose to hate and believe misinformation because that's easier than learning the language"
12:10:49 <zid> I gave you the chance to un-spread it, you refused and told me to join your cult instead
12:10:54 <zid> your credibility with me is absolulute 0
12:11:04 <zid> also my spelling
12:11:27 <klys> are we arguing why zid should learn c++ ?
12:11:35 <graphitemaster> I don't need you to think I'm credible for me to point out how wrong you are
12:11:59 <zid> graphitemaster: I need to think you are if you expect me to do anything you tell me to do or believe anything you tell me.
12:12:05 <zid> Otherwise I'd be a trump supported
12:12:09 <zid> s/ted/ter
12:12:17 <zid> what the hell is up with my capacity to type today
12:12:24 <graphitemaster> honestly with the display of intelligence going on I'm surprised you're not a Trump supporter
12:12:30 <booyah> <3 Trump
12:12:35 <klys> I'm a bit lost
12:12:39 <zid> "The fact you aren't in my cult makes you an idiot"
12:12:58 <klys> zid, is it because you don't know c++ ?
12:13:31 <booyah> graphitemaster: that statment doesn't make sense. It takes some effort to see through bullshit from massmedia and support Trump
12:13:37 <zid> klys: yep, he told me I was stupid for trying to analyze parts of C++ without dedicating 20 years of my life to understanding the divide revealed truth hidden in C++ basically
12:13:56 <graphitemaster> I called you an idiot for not understanding basic communication
12:13:57 <zid> and that the thing I linked pointing out how awkward const was is heresey and unfounded, but can't explain why
12:13:59 <klys> divine*
12:14:04 <zid> yea I seriously cannot type
12:14:09 <klys> k
12:14:16 <graphitemaster> I never called you an idiot for not understanding C++
12:14:24 <graphitemaster> I called the author of the FQA an idiot though
12:14:56 <booyah> lol 20 years
12:15:17 <klys> well, c++ is considered obsolete by some d purists these days
12:15:35 <zid> klys going with the bizzare D is better strategy, let's see how that goes :D
12:15:55 <booyah> c++ is indeed impossible to complety 100% master, but const and such are not that
12:16:01 <graphitemaster> right like we're going to pretend a language which has a mandatory GC to use it's standard library is a better language
12:16:30 <booyah> maybe rust. but tools take like a decade to really mature
12:16:34 <zid> cute, he's in his little pigeon hole, throwing shit at the other pigeons
12:17:02 <zid> "nobody could possibly want a GC for any reason, because C++ doesn't have one", presumably is the logic there, but who can tell
12:17:20 <klys> zid, have you seen akka/scala before? like this: https://github.com/akka/akka-samples/blob/2.5/akka-sample-fsm-scala/src/main/scala/sample/become/DiningHakkersOnBecome.scala
12:17:22 <bslsk05> github.com: akka-samples/DiningHakkersOnBecome.scala at 2.5 · akka/akka-samples · GitHub
12:17:38 <graphitemaster> garbage collectors are never the right answer to any problem, they're a solution to a problem you wouldn't have in the first place if you designed your language not to be stupid
12:17:42 <booyah> http://www.hboehm.info/gc/ A garbage collector for C and C++
12:17:43 <bslsk05> www.hboehm.info: A garbage collector for C and C++
12:17:59 <booyah> but also what graphitemaster said
12:18:07 <zid> klys: I can't say I have beyond hearing about Scala, I'm not a massive fan of non-C-style grammars though so I couldn't really give you an analysis of this
12:18:22 <klys> okay then
12:18:39 <klys> someone interested in language grammars recently introduced me to it
12:18:48 <graphitemaster> this has been proven with languages that actually care about object lifetime and ownership semantics, the only ugly thing you have to deal with is the possibility of causing cycles but even that has some very clever solutions that far exceed the performance of garbage collectors
12:19:11 <graphitemaster> take a look at Rust for instance
12:19:28 <graphitemaster> GC is just lazy
12:19:38 <booyah> even the name says it
12:19:40 <klys> C has had a garbage collector for a long time, called libglib
12:19:44 <zid> I think graphitemaster is coming down on the side of "It is absolutely imperative the language be as terse and hard to write as possible"
12:20:21 <zid> of course, it's hard to tell, he doesn't actually say anything
12:20:32 <zid> He just wants to shout his opinions
12:20:33 <graphitemaster> no, it's absolutely imperative that you don't pay for what you don't use in languages
12:20:46 <zid> that's getting closer, at least
12:20:49 <graphitemaster> this is why C++ is so damn popular, not because it's good, it's actually a pretty shitty language
12:21:15 <zid> I think graphitemaster is coming down on the side of "It is absolutely imperative the language be as terse and hard to write as possible; because someone might lose a cycle if it took 400 hours less to maintain 4 lines of code, per week"
12:21:16 <zid> updated
12:21:53 <booyah> *proceeds to write kernel based on GC*
12:21:57 <klys> c++ introduced us to inheritance and templates. loads of languages, python, java, a lot of newer languages use inheritance. it's practically a given these days that you are oo if your language is new.
12:21:57 <zid> That is precisely not why C++ is so popular, it's popular because it's popular.
12:21:59 <booyah> *realized it was fucking stupid*
12:22:02 <zid> Pop music is all terrible music for a reason
12:22:19 <graphitemaster> Java has a garbage collector and is a language with ugly shit like dependency injection and really simple semantics when it comes to const (everything is immutable if you want to change you must copy everything, change the copy, then reassign) and look at how awful of a mess that shit is
12:22:50 <booyah> pausing your program for a seond to "scan memory" is cool too
12:22:54 <graphitemaster> also 400 hours to maintain 4 lines for sure, spread across 400 files :P
12:23:00 <zid> saying something is good because it is popular has a lovely little latin origin you may have heard of even
12:23:14 <zid> s/of//
12:23:22 <john_cephalopoda> C++ has some nice features over C, like actual proper string handling instead of char pointers and arrays.
12:23:33 <graphitemaster> right, just because there's a latin phrase for something doesn't mean it's relevant
12:23:51 <zid> Disagreeing with 2000 years of philisophy, bold move
12:24:06 <graphitemaster> philosophy is a pseudo-science
12:24:30 <klys> it's a humanity, properly
12:24:30 <john_cephalopoda> Also other sciences have held beliefs for over 2000 years, which were eventually proven to be wrong.
12:24:44 <zid> Yes, we get that he is claiming that, john_cephalopoda
12:24:52 <zid> else he wouldn't have disagreed with it
12:25:43 <john_cephalopoda> Just saying that "Being 2000 years old" is not a sign of quality, neither for a thing nor against it.
12:25:44 <graphitemaster> I don't rely on anecdotal evidence
12:26:05 <zid> despite exclusively giving personal beliefs as fact, bold statement
12:26:28 <zid> the one time you didn't, was when you decided that argumentum ad populum was incorrect to be labelled as a fallacy
12:26:38 <zid> but I guess even that was a personal belief as a factoid
12:27:36 <klys> those are properly claims in epistemology, one of the five branches of philosophy; the study of knowledge.
12:28:06 <zid> klys: you forgot "of incorrect knowledge", apparently it's wrong
12:28:44 <klys> well be more direct. philosophy can't easily be the blanket topic.
12:28:58 <graphitemaster> look if you're just going to cherry-pick my statements and fit them to "your logical fallacy is" then I'm not going to have any constructive conversation with you because you've already made up your mind that nothing I say is going to convince you it's only going to be more fuel for your depressing choice to waste everyone's time by shoe-horning it into your shitty calculator to feel better about yourself
12:29:04 <zid> Sure it can, the philosophy of blankets ;)
12:29:14 <klange> fight fight fight fight!
12:29:24 <zid> graphitemaster: I've been waiting for you to be constructive for half an hour now
12:29:32 <Mutabah> Hold on hold on!
12:29:35 <Mutabah> *grabs popcorn*
12:29:38 <Mutabah> Continue
12:30:27 <zid> I think he quit
12:30:29 <graphitemaster> well I'm not going to give you the satisfaction of being constructive because you've already shown yourself to be unworthy of it
12:30:32 <zid> which works for me, honestly
12:31:43 <klys> the unfortunate thing about c++ seems likely to be the use of templates in derivative languages; and how it makes things less readable generally; from my perspective, someone who avoids and doesn't really know how to use templates anymore.
12:32:18 <zid> klys; I'll play graphitemaster: "They are all mandatory for performance, all tasks need all performance all the time, we should post this to stackoverflow"
12:32:37 <graphitemaster> apart from emotion, there's nothing logical about hating something unless they don't understand or choose not to understand something
12:33:01 <zid> I'm honestly amazed
12:33:14 <klys> I'm only literate with c++ to a point. there is some stuff I just can't handle trying to read.
12:33:32 <zid> He's managed what, 45 minutes without mentioning a single C++ semantic, in a conversation about how C++'s semantics leave room for an exaplanation of the explanation of the semantics of C++
12:33:42 <bcos_> klys: My "language litmus tests" is how hard it is to look at an unfamiliar piece of code and be able to guess where cache misses will be. For C++ you can't even look at "a = b+c" without knowing if someone overloaded the '+' with code that formats your hard drive.
12:33:56 <zid> bcos_: pretty similar here
12:33:58 <klys> aw gee bcos_
12:34:23 <zid> apparently the solution to that is to spend 20 years reading the C++ spec back to front to make sure you're not missing anything
12:34:29 <zid> so that you can lose 5% performance over using straight C
12:34:55 <klys> overloaded operators are still all right in my book, because you can at least use them to try to look like python
12:35:06 <graphitemaster> well that's a silly language litmus test, that's like saying your maths litmus test is to look at an unfamiliar formua and beable to guess where the prime numbers will be
12:35:15 <zid> C++ is a language for creating meta-languages in, as I see it
12:35:26 <zid> You write 800 lines of templates and crap so that your actual program can be a single left shift
12:35:35 <klange> lol
12:35:44 <zid> I am deadly serious about that
12:36:20 <graphitemaster> just sounds like someone who doesn't know how to write C++
12:36:27 <Mutabah> You _can_, but that doesn't mean that you should
12:36:30 <zid> Go look at how anyone actually uses C++, they'll show you their lovely main loop or whatever which is {Type a; f(a); return 0; }
12:36:42 <graphitemaster> "I wanted to do a thing but ended up writing 800 lines of templates and turned my thing into a single left shift"
12:36:45 <zid> and in C that'd just call a function or whatever and you could follow it
12:37:01 <Mutabah> Huh, *mentally looks at work code* Nope, no hacky over-simplified main loop here
12:37:01 <zid> but in C++ the declaration invokes 400 lines of consutrctors and init functions and stuff, the f() is the same, etc
12:37:12 <zid> Mutabah: bwahahah
12:37:20 <klys> programming linguophiles are in a pretty different camp from systems programmers. I rarely hear them refer to c nor c++, and most of the time it's things like lisp or haskell, or something completely new.
12:37:21 <graphitemaster> I can write an entire C program in a macro, I've seen it in GCC source code for christ sake, a single macro that was 800 lines long, your point?
12:37:40 <Mutabah> (I'm mostly serious - There is a little bit of inheritance abuse in the older code, but the new stuff is pretty straightforward)
12:37:56 <zid> graphitemaster: amazing, your first statement about an actual language in 50 minutes, can we have started here instead?
12:38:27 <graphitemaster> I've seen a lot of bad C++ code, if anything I'll agree that it's far too easy in C++ to misuse the language and make a complete fucking mess of it if you don't fully grok how everything is meant to be used together
12:38:39 <graphitemaster> because there's just so much opportunity and ways of doing one thing
12:38:39 <zid> Mutabah: My friend implemented a tetris game by right shifting the playfield class to clear lines
12:38:50 <zid> That's the kind of crap C++ programmers do, and I can't wait until graphitemaster tells me it isn't
12:38:55 <zid> because I'm literally talking about a thing that happened
12:38:57 <klange> the true measure of a language is how effectively you can abuse it
12:39:01 <zid> he'll call them not real C++ programmers or something
12:39:03 <lachlan_s> What's going on?
12:39:21 <klys> summary: c++ is another man's treasure
12:39:38 <graphitemaster> I prefer you only overload operators when they make sense, like if you're implementing a numeric type obviously you'll want +, - and * and / to do their mathematical things (and also be associative and commutative if possible)
12:39:50 <graphitemaster> if you have a collection that is indexable then yes you should overload []
12:39:57 <graphitemaster> or () if it can be indexed by multiple scalars
12:40:12 <graphitemaster> I would never allow code to pass code review if the overload of choice made no sense
12:40:15 <lachlan_s> Oh, for instance, the c++ streaming operators doesn't make sense to me.
12:40:18 <Mutabah> zid: It's a balance between having a powerful and expressive language, and preventing abuse
12:40:35 <graphitemaster> all of iostream is a failure
12:40:42 <zid> Mutabah: yea, I think C++'s balance is waaaaay over on the right wing of "You can do whatever you want, fuck anyone who has to read this"
12:41:05 <graphitemaster> anyways it's not like Rust makes any of this easier
12:41:07 <Mutabah> Possibly... but there are historical reasons for it, and it still does its job of being a useful language
12:41:13 <graphitemaster> in Rust you can implement traits for things that should not implement them
12:41:18 <lachlan_s> It's true, rust lets you do this
12:41:21 <Mutabah> graphitemaster: Well... it solves a whole host of C++'s issues
12:41:22 <lachlan_s> I see less of it however
12:41:24 <zid> Yea it's still useful, but to claim the design of C++ is good is just insanity
12:41:30 <graphitemaster> there is nothing preventing you from implementing a nonsencial trait
12:41:35 <zid> It could do the things it does without being like it is, even though there are historical reasons
12:41:38 <zid> that isn't how it HAS to be
12:41:49 <zid> so to claim it's the platonic ideal at doing what it does? laughable
12:41:51 <lachlan_s> We have stuff like format!(...) instead of `<<` operator overloads because of proc macros
12:42:42 <graphitemaster> one thing I don't like in Rust is how unlike Rust it's hygienic macros are, in C++ templates and constexpr, constexpr if and soon metaclasses are all very much just more C++ that runs at compile time or is generative
12:42:51 <graphitemaster> in Rust the macro system is a completely separate language
12:42:58 <Mutabah> graphitemaster: Sure, but in any language that has operator overloads you can abuse them
12:43:06 <lachlan_s> That is fair.
12:43:18 <Mutabah> ^ fair, and a known problem :)
12:43:18 <lachlan_s> Actually, proc macros are just rust that runs at compile time
12:43:31 <lachlan_s> They're still not quite stable yet
12:43:46 <Mutabah> That said, rust macros are more like a better-integrated version of CPP
12:43:49 <graphitemaster> one other thing I dislike is Rust generics don't allow integers only types
12:43:57 <graphitemaster> makes it difficult to port a lot of my code from C++ to Rust :P
12:44:10 <Mutabah> graphitemaster: In progress, ... just taking while it seems
12:44:20 <lachlan_s> Yeah, that's just because they haven't added an Integer trait to libstd
12:44:22 <graphitemaster> I can point out things I don't like in any language
12:44:32 <zid> I'm not allowed, though
12:44:35 <zid> I get the bible thrown at me
12:45:03 <zid> and get told it's because I don't fully grok the complexities and intricasies of C++ and how wonderful it actually is if you ignore /all the objectively fucking shit parts/
12:45:15 <lachlan_s> graphitemaster: You can just make an Integer trait and implement it for all the integer types
12:45:28 <klys> zid was trying to discuss languages from both epistemic and political perspectives, and you can't be general, accurate, and simple at the same time. GAS principle.
12:45:30 <lachlan_s> zid: Yeah, I get that a lot on reddit
12:46:56 <graphitemaster> in C++ I dislike the standard library, I think it's trash, it assumes exceptions will be used, it lacks any mechanism for which you can inspect the internals (really sucks for hash table and hash sets when you want access to the nodes), a lot of them don't have a non-owning view type (there's string_view but where's vector_view and where's unordered_{map,set}_view), I dislike that you can't have template strings, I'm not a fan of
12:46:56 <graphitemaster> how std::visit is meant to be used with variants, std::any is a mess, I dislike that there's no simple shrink_to_fit operations for other collection types, I'm not a fan of range based for loops (baking begin/end and iterators into your types sucks, have you ever written an iterator, you need to write four of them because there's the const and non-const types, it's boiler plate as fuck)
12:47:24 <zid> lachlan_s: I call it C++ stockholm syndrome, they went to the trouble to waste 5 years of their lives or whatever getting past all the insanity. They couldn't possibly admit that their time was wasted, so the reamining option is to just say C++ is a 'difficult tool to master but worth it' or whatever.
12:47:33 <zid> Or think they got some extremely valueable skill out of it, or some other nonsense
12:47:42 <zid> when in practice, yea, they just had their life partiallyw asted
12:48:25 <Mutabah> Except that knowing C++ gets me paid ;)
12:48:38 <zid> Mutabah: So would knowing how to weld
12:48:42 <graphitemaster> Rust got rid of the ternary operator which pisses me off the most because I like writing short and concise (which often means terse) code
12:48:42 <zid> I'd rather know how to weld
12:49:04 <zid> It turns out welding is actually a skill, and not just a thing you learn to fix other people's welding
12:49:07 <graphitemaster> Rust penalizes you by forcing you to write out the if (blah} { expr } else { expr } mess
12:49:16 <graphitemaster> which I think is less readable
12:49:30 <zid> C++ is a thing you learn so you can fix C++'s design by using templates and operator overloads and exceptions and stuff
12:49:45 <graphitemaster> though often I've noticed that a lot of Rust class types like Optional are smart in that they have stuff like "value_or" which bakes in the small ternary you'd do into the function
12:50:11 <graphitemaster> still think it would be better to just have a short hand
12:50:21 <graphitemaster> no need to remove ternaries because people find them confusing
12:50:39 <lachlan_s> imo, I think removing ternaries was a good idea
12:50:49 <graphitemaster> the try! thing got baked into ? as a suffix on stuff returning Optional that you'd pattern match against and handle Err case
12:50:52 <lachlan_s> Since if statements are expressions in rust, no functionality is lost
12:51:01 <graphitemaster> that's an example of a weird silly hack that may be confusing for people
12:51:11 <graphitemaster> why have that but not ternary I mean every language has ternary
12:51:12 <lachlan_s> It's not a hack though, it's intended usage
12:51:28 <klange> I'm running out of popcorn, can we wrap this up?
12:51:31 <zid> lachlan_s: So like, an if() can have a value, or what?
12:52:04 <graphitemaster> everything in Rust is an expression
12:52:05 <lachlan_s> zid: Like `let c = if 1 == 1 { "a" } else { "b" };`
12:52:13 <zid> so, yes then?
12:52:34 <zid> I'm assuming there c is either "a" or "b" afterwards?
12:52:43 <zid> Does it get weird when mixed with control flow ifs?
12:52:52 <zid> rather than it being used as a cmov there
12:53:03 <graphitemaster> rust doesn't have a return statement, it's the last expression in a block is implicitly a return
12:53:11 <lachlan_s> It does have a return statement
12:53:11 <graphitemaster> in this case c would be "a"
12:53:17 <lachlan_s> Just it's not used much
12:53:20 <MDude> Remove both ternary and if/then operators, use jump tables for everything.
12:53:24 <lachlan_s> ^
12:53:27 <zid> oh so I should read that as return "a" else return "b"? yea sounds a bit like a ternary
12:53:38 <lachlan_s> Yeah, essentially the same thing
12:53:39 <graphitemaster> yeah it's basically a more verbose ternary
12:53:43 <lachlan_s> Just without a special syntax
12:53:50 <graphitemaster> let c = 1==1 ? "a" : "b" reads better imho
12:53:52 <klange> reminds me a bit python ternaries though the syntax for that is kinda backwards, sounds more english-y: "a" if 1 == 1 else "b"
12:53:54 <zid> and you can capture that return value by treating the entire if as an rvalue.. kinda neat I guess?
12:54:05 <zid> I'd have to use it and find out where it sucks to decide if I like it, but it definitely sounds interesting
12:54:48 <graphitemaster> where it gets tedius in Rust is that the { } are mandatory
12:55:00 <graphitemaster> if it was just like C where single statements could omit the { } it would be less annoyinh
12:55:09 <graphitemaster> you could write "let c = if 1 == 1 "a" else "b"
12:55:20 <zid> bugs with that were solved by linters 40 years ago, I don't mind {} elision
12:55:29 <lachlan_s> The single statement without the brackets always annoyed me
12:55:41 <zid> but I guess when it's an rvalue like that.. *with* brackets is probably better
12:55:55 <graphitemaster> I mean it's probably ambiguous
12:56:07 <graphitemaster> I think the reason if in C/C++ need the parens is because {} is optional
12:56:07 <zid> like, let c = if "a" else "b", d = 12; or whatever (if possible) would suck
12:56:39 <klange> c = "a" if foo == bar else "b"
12:56:44 <graphitemaster> python
12:56:47 <klange> yes
12:56:57 <zid> klange literally copy pasting from other lines
12:57:10 <graphitemaster> also Lua
12:57:21 <zid> god, now lua is a language that sucks in design
12:57:32 <graphitemaster> lua is more about using and/or though
12:57:35 <graphitemaster> x = a and b or c
12:57:39 <graphitemaster> you can do that in C too
12:57:44 <lachlan_s> what does the line even mean?
12:58:02 <zid> It has a whole bunch of stuff that is perfectly fine, then they ruined it in several key ways
12:58:05 <graphitemaster> print(x < 0 and 'negative' or 'position')
12:58:13 <graphitemaster> print(x < 0 and 'negative' or 'positive')
12:58:48 <graphitemaster> would print negative if x is < 0 or positive if x >= 0
12:58:57 <lachlan_s> Ah
12:59:02 <lachlan_s> That's quite confusing syntax
12:59:07 <lachlan_s> the "and" is really not doing it for me
12:59:28 <zid> that.. is not how english works, heh
12:59:35 <graphitemaster> well you can do it in C too
12:59:44 <zid> 'thus' is sort what they need there, but even that is kinda wrong
12:59:48 <graphitemaster> printf("%s", x < 0 && "negative" || "positive")
01:00:04 <graphitemaster> except it will evaluate wrong
01:00:08 <zid> so you can't, then
01:00:17 <graphitemaster> because stupid non-Perl expressions
01:00:32 <klange> yeah I'll stick to ternaries, as weird as their syntax is
01:01:12 <zid> ternary *syntax* isn't weird, but just the concept of implementing things like a venn diagram on a line of text is fucked up in princple
01:01:27 <graphitemaster> I never found them that confusing, even chained ternaries
01:01:45 <graphitemaster> what is confusing is weird value expressions like Lua
01:01:56 <zid> "if a then "a" if b then "b" but if a and c then "c" is just a venn diagram, writing that down in *any* form of language onto a single line is going to be weird
01:02:22 <zid> punnett square operator is never going to be pretty :p
01:02:27 <graphitemaster> ternaries are binary though, you only get one of two out of an expression
01:03:02 <lachlan_s> Oh, in rust, you can do matches in an expression
01:03:11 <graphitemaster> right, you can also do matches inside matches
01:03:15 <lachlan_s> Essentially a ternary with an arbitrary number of values
01:03:31 <graphitemaster> I hate nested pattern matching in Rust, it's so verbose
01:03:49 <graphitemaster> at least in Rust you can drop some of the braces with =>
01:04:13 <zid> c = x ? a : b; -> c = !!x * a + !x * b;
01:04:14 <zid> perfect
01:04:19 <lachlan_s> Yeah, I'll agree with that, there are some weird parts of pattern matching in rust
01:04:21 <klys> print( x < 0 and "neg" or and x == 0 "nil" or "pos" )
01:04:38 <graphitemaster> or and ?
01:04:49 <klys> yeah that's what I'm saying it's weird
01:04:55 <graphitemaster> that's not valid Lua
01:04:57 <zid> printf("%s", (char *[])({"nil", "pos"})[x < 0]);
01:04:59 <klys> k
01:05:03 <zid> obvious way to rewrite that to make sense
01:05:24 <lachlan_s> never do that in real code
01:05:42 <graphitemaster> zid, no the obvious way is printf("pos\0nil"[3*!!(x < 0)])
01:05:58 <zid> That.. doesn't work
01:06:12 <zid> that expression has type char
01:06:23 <graphitemaster> oh oops
01:06:34 <zid> also I think you meant 4
01:07:00 <graphitemaster> printf(&[4*!!(x<0)]"pos\0nil")
01:07:08 <zid> Much better
01:07:10 <graphitemaster> :D
01:07:20 <klange> now you're thinking with pointers
01:07:22 <lachlan_s> how do you delete someone else's comment
01:07:30 <klange> you can't delete anything on irc
01:07:40 <lachlan_s> I know, it was a joke
01:07:50 <zid> lachlan_s: Hi, meet klange
01:07:53 <klange> you're a joke
01:07:58 <lachlan_s> no u
01:08:06 <klys> ur dawg
01:08:32 <graphitemaster> printf("pos\0nil"+4*!!(x<0))
01:08:34 <graphitemaster> is how I'd write it
01:08:37 <zid> https://github.com/zid/boros/blob/master/boot/print.c#L74 I actually used the meme [] semantic trick in 'real' code because I am awful
01:08:39 <bslsk05> github.com: boros/print.c at master · zid/boros · GitHub
01:09:51 <lachlan_s> this is why I use rust
01:09:58 <zid> so that you can meme?
01:10:09 <lachlan_s> nah, so that I can meme outside of coding not inside of it
01:10:18 <zid> but you're using a meme language full of memes
01:10:22 <graphitemaster> Rust code sucks for code golfing
01:10:24 <zid> That's like C's singular meme trick
01:10:40 <zid> You literally just showed you get to write inline if statements as rvalues
01:10:52 <lachlan_s> But you merely adopted the meme; I was born in it, moulded by it.
01:10:56 <klange> psh, C has the best memes, like duff's devices
01:11:21 <klys> mnemonic
01:11:26 <zid> mnemne
01:11:31 <zid> 'meem'
01:11:37 <zid> There's my English meme for today
01:11:53 <graphitemaster> C has a lot of meme's a[b] == b[a] == *(a+b) == *(b+a), and a->b == (*a).b, also duff's device, use of static/const inside array brackets for function prototypes like "void foo(int a[static 3])" oh and my favorite is actually defining structures in functions like "void foo(struct { int x; } a)"
01:12:09 <zid> ' does not mean "here comes an s"
01:12:27 <graphitemaster> oh and multi character literals with ' like 'hell'
01:12:35 <graphitemaster> because literal's in ' have type int
01:12:45 <zid> correctly have type int :P
01:12:53 <lachlan_s> C's type system is a meme
01:13:06 <graphitemaster> bitfields can be fun, especially nameless members
01:13:12 <graphitemaster> struct a { uint32_t : 16; };
01:13:14 <graphitemaster> wat?
01:13:26 <zid> bitfields are wank they don't give you enough control
01:13:34 <graphitemaster> they're fun to confuse people though
01:13:40 <klys> so dig out the reference document, locate the part of the grammar you want, and paste it on a jpeg
01:13:44 <zid> all code-gen through them is basically unspecified
01:13:56 <graphitemaster> taking the address of a bit field is UB too
01:14:05 <zid> so you might as well have just used pre-standard ones where only gcc supported them because then at least you know the semantics are "what gcc does"
01:14:15 <zid> not, "or the fucked up thing msvc does"
01:54:49 <burzos> Why do exec need to be implemented in the kernel? I already have an address space, so can't I just load the new process, maybe load some libs and then put RIP at the entry point.
01:54:57 <burzos> Why does exec*
01:55:16 <zid> it doesn't
01:55:17 <zid> see: pthreads
01:55:23 <zid> but 'exec' and pthreads aren't the same
01:55:31 <zid> for example, when a crash happens, etc
01:55:57 <Mutabah> Sure, you can implement `exec` in userspace if you want. All you need is to ask the kernel to update the page mappings correctly
01:56:38 <burzos> Ok, but all the unix-clones do it in the kernel, so they have some reason?
01:56:43 <zid> You'll only have one pid, that's the difference between kernel and userspace threads/processes
01:56:51 <zid> so debuggers will have to debug EVERY program at once, etc
01:57:22 <burzos> Right, I could allocate a basic process in the kernel, get it a new pid, then move the other stuff into a usermode impl.
01:57:30 <Mutabah> burzos: Simplicity - the kernel already (likely) has an executable loader, so it's easier for the kernelt to wipe the address space and rebuild it based on the executable
01:57:32 <zid> they will have to fight for memory space (they will all be in the same one), etc
01:57:37 <clever> Mutabah: but how about un-mapping the old process that was loading the new one?
01:58:00 <Mutabah> Note - `exec` just replaces the current process's code/data with a new set
01:58:05 <Mutabah> (it doesn't create a new process)
01:58:11 <burzos> Oh, right
01:58:14 <burzos> Yeah, that's forks job
01:58:21 <Mutabah> In clasic unix, yes.
01:58:23 <zid> I took it that he meant execvp or whatever the one that isn't does
01:58:26 <zid> I hope that's right
01:58:30 <Mutabah> (Modern systems use the `spawn` family of syscalls)
01:59:36 <burzos> The kernel uses it's executable loader for something other than loading user space processes?
01:59:42 <zid> its
01:59:54 <zid> erm, no?
02:00:00 <zid> weird question
02:00:15 <burzos> Ok, then saying "the kernel already has an executable loader" doesn't mean anything.
02:00:28 <zid> burzos: someone had to load init()
02:00:33 <zid> in order for it to want to execute another program
02:00:43 <zid> so yes, the kernel already has an executable loader
02:00:53 <zid> err 'init' not init()
02:01:00 <zid> init=/bin/bash or whatever
02:01:12 <Mutabah> (That said, it's fun to do tricks to avoid that :D)
02:01:20 <burzos> That feels like a not good reason.
02:01:43 <zid> burzos: You had this problem last time, you need to change the way you approach these.
02:01:45 <zid> Consider:
02:02:02 <Mutabah> To be honest, it isn't really the best idea IMO - My current OS project doesn't have the ELF loader in the kernel
02:02:07 <zid> Let's pretend the kernel doesn't load programs directly, and instead delegates it to userspace, but userspace acts the same otherwise
02:02:19 <zid> So what would your kernel need instead?
02:02:25 <Mutabah> (it lives in userspace, in a tiny stub that is by-convention mapped at the top of userspace)
02:02:33 <Mutabah> zid: A very simple init-only loader
02:02:35 <zid> It would need everything an ELF loader would need but as a syscall, so loading memory ranges, setting page permissions, creating new address spaces, etc
02:02:48 <zid> those all need to be syscall interfaced now
02:02:51 <Mutabah> Oh, sure, but that already needs to exist for general process management
02:02:54 <zid> when previously none of that was actually required
02:03:13 <zid> I don't even think linux can let you spawn blank address spaces, Mutabah
02:03:20 <Mutabah> ... no, but mine can :)
02:03:36 <zid> You have to load a fake program suspended then wipe it and do complicated ioctls and syscalls to adjust the mappings and stuff to do what you want
02:03:50 <zid> (I looked into it for essentially making a save-state feature like an emulator for processeses)
02:03:56 <Mutabah> Well, you spawn a limited copy of the current AS (copying a fixed range) then set the new thread running with a fixed IP/SP
02:03:57 <clever> related, is packers like UPX
02:04:05 <zid> recreating the old address space requires you to make an ELF witht he same memory layout as you wanted
02:04:09 <Mutabah> zid: Note - I'm saying the way it _can_ be done
02:04:12 <clever> those programs will take an executable, compress it, and prepend it with an unpacker
02:04:13 <Mutabah> Not how linux does it
02:04:24 <clever> and the unpacker will then uncompress the blob, fix up the mappings, and execute it
02:04:32 <clever> so its sort of doing the dynamic linkers job, in userland
02:04:56 <burzos> Mutabah: Did doing it this way, with the simple kernel mode loader, and the more complete usermode loader, reduce the amount of compexity in the kernel?
02:05:38 <Mutabah> Yes.
02:05:40 <zid> I doubt it
02:06:09 <Mutabah> it also removed the duplication of the executable handling (because userland is the only place that handles ELF structures - both the program and dynamic tables)
02:06:39 <zid> That 'executable handling' now just becomes syscalls rather than stuck in elf.c
02:06:57 <burzos> Sounds kind of nice, especially since the execve() codepath is so crazy on linux/netbsd/etc. I'm less worried that a random program is going to find a kernel exploit during startup.
02:07:16 <zid> It's crazy because process management is crazy in general
02:07:26 <zid> if you move it to userland now every userspace process has that crazy code in it
02:07:37 <burzos> Which is wayyy better than it being in the kernel.
02:07:40 <zid> and the kernel has an 40 interfaces to all that crazy mess
02:07:43 <zid> all which can be buggy
02:07:56 <burzos> 40 simple interfaces are easier to get right then 3 crazy ones.
02:08:01 <Mutabah> zid: They're already going to have the dynamic linker
02:08:15 <zid> you've turned if(p->x == E_BLAH) p->x |= E_FUNC; into sys_get_x_of_p(p); sys_add_flag_to_p(x); etc etc
02:08:30 <Mutabah> zid: And for debugging you'd already have a way of manipulating remote address spaces
02:08:37 <zid> with all the user/virt pointer checking and stable ABI crap that introduces
02:08:56 <zid> user/kernel*
02:10:16 <zid> Mutabah: Linux/windows do not.
02:10:39 <Mutabah> zid: Not have a way of manipulating remote processes? I'm pretty sure the windows debug API has that
02:10:58 <zid> processes? yes. Address spaces? no
02:11:07 <zid> none of those syscalls overlap with process creation though
02:11:11 <Mutabah> (Huh, oh yeah, I don't even do the remote manipulation)
02:11:22 <Mutabah> (the loader stub just gets given a path to load, and it does it itself)
02:11:24 <zid> you'd need *additional* address space manipulation syscalls to not have a loader
02:12:14 <geist> yah we jump through some amount of hoops in zircon to avoid that
02:12:17 <Mutabah> https://github.com/thepowersgang/rust_os/blob/master/Usermode/loader/src/interface.rs#L26 - `new_process` and `start_process` are the API calls, `new_process_entry` is the new process's entrypoint
02:12:19 <bslsk05> github.com: rust_os/interface.rs at master · thepowersgang/rust_os · GitHub
02:12:35 <geist> kernel has no loader, what it has is an initial process that is a flat binary and must be mapped into process 0
02:12:50 <geist> and then from then on out user space decides how it wants to bootstrap the next thing
02:13:18 <zid> You'd need like: Create empty address space, allocate stack into black address space, load section as text into remote address space, giveremote processes security tokens they don't have, and like 2000x other things to be able to interact with a remote process like it was your own process so that you can arbitrarily create a process in any state - which would otherwise just have been flags in
02:13:19 <zid> your executable format
02:13:41 <geist> that's what zircon does, and it's far less than 2000x other things
02:13:44 <geist> it's like 3 things
02:13:57 <zid> geist: Right, obviously mine is for something like a real OS people use ;)
02:14:01 <zid> *runs from thrown shoes*
02:14:06 <burzos> That's cool. What were your reasons for implementing it in Zircon that way?
02:14:08 * geist throws shoe
02:14:16 <Mutabah> zid: I take back what I said about needing remote AS manipulation, you just need to take a minimal clone and hand it handles
02:14:33 <geist> burzos: well, it's not much different from what a lot of unices do with 'user space loaders', like ld.so
02:15:01 <zid> Mutabah: bear in mind the 'clone' syscall has to work on the modified *second* process
02:15:01 <geist> but since zircon is a microkernel, you have even less things that the kernel can do, like know what a file is, or what ELF is, or how a process even wants to start
02:15:08 <zid> so the kernel needs making aware of anything important you did
02:15:12 <Mutabah> zid: Sure, but that's in the kernel
02:15:12 <geist> what even run time its implementing (posix, Go, win32, whatever)
02:15:42 <zid> maybe what burzos actually wants is a microkernel
02:15:50 <zid> all his questions start with the supposition that the kernel should be doing less shit
02:15:59 <geist> thing is that it's not much more so than what something like linux does
02:16:08 <Mutabah> ... wasn't burzos arguing for having an in-kernel malloc yesteday?
02:16:11 <geist> in general the kernel is only responsible for getting the user space loader binary in the process, ld.so
02:16:23 <geist> and then from then on out all the complex dynamic linking is done in the context of the new process
02:16:26 <Mutabah> geist: Random request, what do you think of the loading model in my code above?
02:16:29 <geist> and thus operates on its own address space
02:16:37 <Mutabah> (Sorry about the slightly noisy code)
02:16:48 <burzos> Right, so you've just extended the function of ld.so
02:17:12 <geist> Mutabah: seems okay i guess, but i dont speak Rust
02:17:35 <Mutabah> No biggie, don't feel obliged to look too closely
02:17:46 <zid> geist: Is it like, remotely possible to make something like an irc bot sandbox that compiles programs like candide/jeordi on a microkernel?
02:17:54 <geist> you should ask that tPG guy about it though
02:18:00 <geist> zid: why not?
02:18:08 <geist> it's just a user space concept, not a kernel concept
02:18:23 <geist> so things like 'where the file system is' is entirely based on you having handles to the appropriate user space services
02:18:25 <Mutabah> geist: Ha. ha. ha.
02:18:35 <zid> right but the reason I can't start an arbitrary process or not corrupt the filesystem or whatever inside candide is because the kernel stops it
02:19:14 <geist> right, in the case of zircon it's up to the particular process to have sufficient handles to be even able to see a file system, and/or create new processes
02:19:25 <zid> what's stopping me stealing handles, guessing handles, etc
02:19:29 <zid> and bricking your filesystem
02:19:35 <geist> so in the case of something like the irc bot you'd likely run the bot in a sufficiently sandboxed job tree that has no rights
02:19:38 <burzos> geist: So if I want to control the network interfaces available to a process, this is done with a contract to a usermode process?
02:19:46 <geist> handles are 100% private to the process
02:19:54 <Mutabah> zid: Kernel-controlled handles I assume (like file descriptors, except more general)
02:19:55 <geist> burzos: correct
02:20:16 <zid> Mutabah: But that doesn't sound very microkernelly to me honestly, if you have shit loads of handle tracking and crap in the kernel, shrug
02:20:30 <burzos> Yeah that's great, I think it tells a much better story about how to containerize the userspace.
02:20:30 <zid> it just changes what shape the kernel is, doesn't really change how big and complex it is
02:20:36 <geist> zid: if what you mean is the process that you compiled and are running, then you'd almost certainly start it up without access to network/filesystem handles
02:20:49 <geist> and in that case you can't guess anything, because the handles dont exist when the process was started
02:21:00 <zid> yea someone already mentioned "just imagine fds", that works fine
02:21:06 <zid> as an explanation
02:21:27 <geist> yah the general difference with capability based models is a process has no intrinsic rights to do much of anything
02:21:49 <geist> it has to ask for it based on already having a handle to something that was bequeathed it with sufficient capability to do something
02:21:50 <zid> I like ACLs and all that crap, but putting all that code into the kernel just sounds like a normal kernel to me :P
02:21:54 <geist> either at birth or later
02:22:15 <geist> ACLs are not exactly the same thing at all
02:22:30 <geist> they're a method that you may use, but not really the equivalent of a capability. they'r emore of a tool
02:22:37 <zid> 'and all that cpra
02:22:52 <zid> you won#t let me play fast and loose ;(
02:22:55 <zid> wow, typing
02:23:03 <geist> much typing very wow
02:23:06 <zid> I think I suddenly have parkinsons
02:23:44 <burzos> geist: You guys have an explicit model of the different resources that can be allocated to a given process?
02:23:51 <geist> yes
02:24:06 <burzos> Yeah that sounds awesome
02:24:20 <geist> for the most part, there are some holes in it that i'm thiking about
02:24:38 <geist> one of the downsides of a model like that is there's no intrinsic ownership of anything, that the kernel can track
02:24:57 <geist> a handle to a kernel object doesn't make the object 'yours', since others can have the same handle
02:25:25 <geist> so it's hard to limit the amount of something a process can do, short of giving them a binary YES/NO that you get with a capability
02:25:37 <geist> as in 'can you make a thread or not?' or 'can you map memory into your process or not?'
02:25:48 <geist> but it's much harder to limit say 'you can have 8 threads but not 9'
02:26:20 <ekleog> geist: oh curious, do you have some benchmarks of zircon vs. linux, just to get an idea whether the microkernel approach actually does or does not impact performance in a meaningful way due to context switches?
02:26:33 <johnjay> ekleog: that sounds cool. science rox
02:26:48 <geist> not particularly. in general linux is going to stomp zircon at things that linux likes to do
02:27:03 <geist> lots of process creation/teardown, file io
02:27:19 <zid> micros seem like an okay way to be 'hands off' if a lot of what you need to do is just in a long-standing process with lots of privledges
02:27:23 <geist> those are things that linux and other posix oses 'own' at
02:27:43 <ekleog> (also, hello all, I think I came here a few years ago when I was doing a toy os for understanding how things worked, and then completely forgot about this chan until seeing it again a few days ago :))
02:27:52 <geist> our general zircon model is lots of little processes with limited priviledges
02:28:01 <geist> ekleog: welcome back!
02:28:05 <geist> we're still here
02:28:15 <zid> What's an goelke
02:28:29 <zid> oh a leo g
02:28:51 <ekleog> yup ^^
02:29:05 <ekleog> no real meaning
02:29:16 <burzos> geist: The problem with the handles is mostly about handles that are given to you by another usermode process, not the handles that I directly allocate.
02:29:44 <ekleog> geist: hmm, so there's no comparison for “useful” stuff? I guess it's hard to define “useful”, then, anyway, and the programs for microkernels won't be written in the same way as for monolithic ones :/
02:30:40 <ekleog> maybe something like “running a web browser” could be a good real-world test, but… well…
02:31:04 <zid> I doubt web browsers are spamming syscalls enough for you to measure any difference honestly
02:31:21 <Shockk> random quick question; is it going to be faster on x86 for me to swap two unsigned chars by 1. using a temporary 2. using the three xor method?
02:31:34 <ekleog> well, if you don't measure any difference on a web browser, maybe it means the performance impact isn't meaningful and could be neglected :)
02:31:37 <zid> using a temporary is going to be better on *every* architecture
02:31:41 <zid> it depends what the cost of your temporary is
02:31:48 <geist> burzos: oh you can make your own handles to things, and you can give them away
02:31:51 <Shockk> hmm okay
02:31:52 <zid> if that means "I have to order another hard drive on amazon" that may be slower
02:32:06 <geist> so you could, for example, claim to be some sort of video decoder service, load a bunch of decoder codecs
02:32:07 <zid> if it means "I will just use a spare register" that will actually be 0 cost
02:32:11 <zid> because of the register renamer
02:32:21 <Shockk> it most likely does mean that
02:32:21 <ekleog> zid: “unsigned chars”
02:32:24 <geist> then advertise your service with some sort of index that others can go find
02:32:40 <zid> ekleog: I can read
02:32:57 <ekleog> oh, you replied wider than the question, then :)
02:33:03 <zid> nope
02:33:08 <zid> chars fit in registers just fine
02:33:32 <ekleog> Shockk: there's the xchg instruction, if you really want perf, I guess
02:33:35 <geist> but large standard services that many processes may get a handle to and/or go request via asking a service for it is say the network stack or some sort of local file system
02:33:53 <Shockk> ekleog: I know about that, though can't use that in portable C of course
02:34:18 <zid> Shockk: You're basically REQUIRING that the C compiler recognises the xor swap and optimizes it away
02:34:36 <zid> because it's guarenteed just generating different code is just better
02:34:40 <zid> oh for fuck sake who sent him here
02:34:49 <zid> I want names
02:35:02 <geist> xor swapping may or may not be as efficient on a modern machine, since a mov to temp and then mov back the internal scheduler may be able to optimize away
02:35:21 <Shockk> ah right
02:35:35 <zid> if xor swap is best, trust the compiler to generate it
02:35:39 <Shockk> hm okay
02:35:42 <geist> an xor based swap is a series of interrelated instructions that the scheduler may need to run back to back, but a series of movs to temporary and back it can probably do in parallel
02:35:49 <zid> (it won't be)
02:35:57 <burzos> I'm trying to understand why the kernel can't track ownership of handles. If the process allocates the handle, then the kernel knows what it is and how many I can have. If someone else sends me the handle, the kernel will see the IPC transmission and can make a similar decision.
02:36:18 <ekleog> the cost of mov-based swap is still that it consumes one register, though, so if you're starved for registers it may mean spilling
02:36:20 <geist> burzos: sure, but if i duplicate a handle and give a copy to someone else, do i still 'own' it?
02:36:28 <geist> what if i then throw away my local copy
02:36:30 <ekleog> (anyway the compiler will know better)
02:36:32 <geist> do they own it?
02:36:35 <burzos> You lose ownership after an explicit release
02:36:39 <zid> ekleog: spilling, not xor swapping, you said it yourself
02:36:41 <Jmabsd> if I make a program that I run on bare metal - say a computer game or a word processor - and this program is made up of a ton of submodules that *not trust each other* but have their own memory privileges - how fast will my program work?
02:36:46 <Jmabsd> the management logics in my game/word processor would have some glue to switch memory privileges when jumping from one privilege zone to another. there might be, say, 1,000-10,000 CPU instructions of work between jumps, and access of, say, 100-1,000 memory pages, between jumps.
02:36:49 <Jmabsd> rephrased: my computer game, which runs on AMD64 bare metal, has submodules A .. Z . there are multiple memory access privilege contexts, for simplicity let's say each submodule has its own single context. each malloc:ation has its own memory privilege setting, which has a RWX setting potentially for each context, so potentially A .. Z separate privilege settings for each mallocation. when jumping from one submodule to another, what
02:36:53 <Jmabsd> will be the cost of the memory privileges switch?
02:36:53 <zid> here it comes
02:36:54 <geist> sure, but then how do you limit it? dont let it transfer in the first place?
02:37:16 <geist> Jmabsd: some.
02:37:19 <zid> Jmabsd: The answer didn't change inbetween 4 minutes ago when you asked me, and 3 minutes ago when several people told you to fuck off
02:37:28 <Jmabsd> geist: what do you mean?
02:37:29 <zid> You don't know what any of those words mean and you refuse to accept what anyone tells you
02:37:31 <geist> the answer is, yes. there's overhead
02:37:34 <burzos> If the receiving process has too many of handle X, then the kernel can fail the send.
02:37:59 <Jmabsd> zid: noone told me to fuck off on #asm. this is a valid and general question. i do need to read Intel's developer manuals however this is such a general question that someone may have an idea already.
02:38:03 <geist> burzos: therein likes to rub. now yo uhave a much much more complicated IPC
02:38:09 <geist> you just increased the cost of it substantially
02:38:17 <zid> That it's a general question is precisely WHY you were told to read the manual
02:38:20 <zid> It has no answer
02:38:23 <geist> zid: let them ask.
02:38:25 <zid> YOU need to research what your question is
02:38:40 <geist> we may arrive at the same answer, but that's for us to decide
02:38:49 <zid> go for it, he's mental, glhf
02:39:00 <geist> duly noted
02:39:18 <ekleog> <zid> ekleog: spilling, not xor swapping, you said it yourself <- I'm not sure I understand… the mov-swap risks spilling, the xor-swap definitely has additional dependencies, which one is better will depend on the surrounding function? (and so we agree letting the compiler pick the best one is best)
02:39:34 <Kazinsal> We've had enough of the mental here to know who's actually mental and who just doesn't understand the concept I think
02:39:35 <zid> ekleog: It will never ever pick the xor swap
02:39:53 <zid> L1 is significantly faster than tying up 2 registers and the alu
02:40:10 <Jmabsd> geist: what do you mean by "how do you limit it"?
02:40:12 <geist> with a sequence of interrelated instructios that it cant run in parallel
02:40:19 <geist> Jmabsd: I didn't say that
02:40:23 <burzos> geist: But this slow path is only hit when transferring handles, or is that 'most' of the IPC
02:40:25 <Jmabsd> zid: conceptually I know very well what i'm talking about.
02:40:34 <zid> Jmabsd: Talk to them, not me
02:40:36 <zid> I've written you off.
02:40:43 <Kazinsal> jesus dude
02:40:45 <geist> for fucks sake man. just drop it
02:41:04 <zid> That was literally me trying to make him stop talk to me
02:41:05 <zid> :/
02:41:22 <Jmabsd> lol. no worries i will not talk to you again before reading the Intel manual chapters.
02:41:32 <geist> sigh. i'm done, i'll come back when everyone has cooled off
02:41:35 <ekleog> zid: hmm the dependencies of the xor swap is like less than two pipeline runs (which will be used for running ahead other code), you think that's more than a L1 access?
02:42:05 <ekleog> (I don't have numbers in my head)
02:43:30 <zid> ekleog: we'd need to burn a register no matter what, the question is just whether we use 2 movs or 2 xors really
02:44:07 <Shockk> hmm so:
02:44:09 <Shockk> "On modern CPU architectures, the XOR technique can be slower than using a temporary variable to do swapping. One reason is that modern CPUs strive to execute instructions in parallel via instruction pipelines. In the XOR technique, the inputs to each operation depend on the results of the previous operation, so they must be executed in strictly sequential order, negating any benefits of instruction-level
02:44:09 <Shockk> parallelism.[4]"
02:44:32 <bcos_> Erm
02:44:40 <bcos_> Someone's forgetting about register renaming here
02:45:02 <ekleog> zid: hmm… I haven't done trickery like this for a while, but isn't xor-swap xor eax,
02:45:04 <bcos_> "2 movs" adds up to "2 * nothing because those registers got renamed instead"
02:45:19 <ekleog> zid: hmm… I haven't done trickery like this for a while, but isn't xor-swap: xor eax, ebx; xor ebx, eax; xor eax, ebx ?
02:45:25 <ekleog> bcos_++
02:45:28 <zid> ekleog: right, three xors
02:45:40 <zid> bcos_: Oh, I mentioned it earlier, so I meant it like "two movs which are free"
02:46:02 <zid> at worst we have to hit L1 a single time because we're out of registers
02:46:17 <geist> note there have been plenty of architectures that have a straight register swap, but for whatever reason i dont think any of the standard microprocessors picked it up
02:46:33 <Shockk> seems xor swap also has a problem that if x and y are the same, then the triple xor will result in 0
02:46:35 <geist> presumably because early on taht would have involved needing either an internal temporary buffer, or two internal busses that they didn't have
02:46:41 <zid> geist: It only *really* matters when you're dealing with mul / div and calling conventions I imagine though
02:46:46 <bcos_> Of course what happens when a value is in memory and not in a register is different - register renaming won't help, and sadly you can't "xchg" without an implied (expensive) lock
02:47:00 <Shockk> so xor swap also requires a condition check
02:47:02 <zid> where things have to be in certain places, rather than just swapping your CODE to target the opposite regsiters
02:47:06 <Jmabsd> does anyone have any thoughts on my question re. the cost of memory access privilege switches happening *often*?
02:47:27 <geist> of course with register renaming a swap registre instruction would be trivial, but then i guess nothing really needs it. seems like arm64 would have added it or something if they had though it was useful enough
02:47:31 <bcos_> Jmabsd: This is easy - "total cost = average cost * how often"
02:47:38 <geist> yep
02:47:39 <ekleog> zid: and so you think hitting L1 once (assuming register renaming can't occur for some reason) isn't worse than two dependencies between the three xors?
02:47:46 <zid> massively?
02:47:57 <zid> l1 is also the *worst* case
02:48:08 <zid> and probably won't ever have to generate
02:48:31 <ekleog> well, the compiler could know whether it'll be the worst case or not and decide to generate one or the other
02:48:33 <geist> mos tlikely there's a third architecture register it can use. the xor trick is very handy if you have some tight code and no other register
02:48:47 <geist> if you have to spill to ram it's then fairly debatable if the non xor version is a good idea
02:49:00 <zid> yea, that's what the debate is, geist :D
02:49:08 <ekleog> ^
02:49:21 <zid> I vote hitting L1 is faster, he votes xor is faster
02:49:23 <geist> yeah, my guess is it's basically never worth doing the xor thing, since you probably can scrounge up a free register
02:49:33 <zid> but I also mentioned I bet it never happens in practice
02:49:40 <geist> if anything bceause you almost always need some temporary register throughout the function
02:49:49 <zid> so doggedly doing the xor to save a register is bad
02:49:49 <geist> so you can just use it here for this, then a few instructions later use it for something else
02:49:58 <ekleog> yeah, we agree it's likely not going to hit this case :)
02:51:19 <Shockk> in case you're all curious, the only reason I was asking is because I had to implement qsort
02:51:32 <geist> then expand it out a little. what if you're referring to an SSE or an AVX256 or 512 register, then the desire to not spill is huge, and the win of not tying down an expensive ALU is a big win
02:51:42 <Jmabsd> bcost_: "how often" = once each ~~~5,000 CPU cycles. do you have any idea of "average cost"?
02:51:49 <geist> register renaming for mega registers like that is a gigantic win all around, plus you have a lot more of them
02:51:59 <geist> Jmabsd: answer is no. it's complicated
02:52:09 <Shockk> oh wait
02:52:21 <geist> i'm not entirely sure what you're looking for. context switching is fairly expensive. but the alternative is not having it at all
02:52:21 <zid> <Shockk> oh wait
02:52:23 <zid> uh-oh
02:52:25 <Shockk> lol
02:52:29 <zid> he's about to tell us he didn't need to swap afterall
02:52:34 <Shockk> does the situation change if the two values I'm swapping aren't registers?
02:52:37 <geist> so since the alternative isn't really that feasiable nowadays, it's just part of the overhead of having a modern machine running a modern OS
02:52:43 <Shockk> i.e. if they're elements in arrays
02:52:54 <geist> and it's part of the OS developer's job to try to mitigate it somewhat
02:53:01 <Jmabsd> geist: a strapped context switching model where you preserve page tables (the whole OS has one single huge page table) and just switch memory accesses (read/write/executability), would that make things cheaper?
02:53:20 <geist> trouble is something complicated like context switching the MMU varies tremendously based on how much TLB it has to dump, how much L1 cache may not be usable now, etc
02:53:22 <Jmabsd> disabling stuff like FPU registers, would that make it cheaper? where can costs be cut on amd64 and generally?
02:53:29 <geist> it may be pretty cheap, a few hundred cycles. it may be 10000 cycles
02:53:32 <zid> Shockk: it just changes whether the xor targets memory and a reg or a reg and a reg
02:53:34 <zid> I think..
02:53:42 <zid> I was too lazy to design out both sets of code in myface
02:53:46 <johnjay> to evaluate how good an OS kernel is do you have to make the entire userland?
02:53:49 <johnjay> or can you benchmark it?
02:54:00 <Shockk> hmm but in the case of the temporary, it wouldn't be able to just do a register rename right?
02:54:03 <geist> yes and yes. it's all tradeoffs. if you dont dirty the fpu regs then there's less state to save. my experience is in the 50-100 cycle range for simple SSE xsave
02:54:04 <milesrout> depends what you mean by 'good'
02:54:10 <geist> probably scaled up appropriately for AVX256 and 512
02:54:22 <johnjay> milesrout: by metrics presumably
02:54:29 <Jmabsd> geist: interesting. i have a question about TLB flushing which is the most expensive part - is it expensive becuase of any interaction between CPU cores, or it's expensive just because it's a lot of data processing locally within the CPU core?
02:54:34 <bcos_> Jmabsd: If "context switch" means changing page table permissions (RWX flags) and then invalidating all of them; it'll be horribly expensive due to the TLB misses (especially if you compare it with "different virtual address spaces and PCID/address space IDs")
02:54:44 <milesrout> you seem to mean 'fast' but even then it depends what you mean by 'fast': low latency? high throughput? and of what? network traffic? disk traffic? pure computation?
02:54:45 <Shockk> hmm here let me gist what I've got so I'm not struggling to ask about it
02:54:50 <milesrout> what metrics
02:55:10 * geist sheds a tear, suddenly we have like 3 conversations gooing on simultaneously
02:55:20 <geist> sometimes we get zero for days on end. must be that time of year (schools in!)
02:55:21 <Jmabsd> bcos_: are you aware of any article or compendium discussing this?
02:55:21 <zid> hmm I think you'd need two regs to not xor actually? heh
02:55:28 <milesrout> i don't think an operating system is best measured by purely it's performance anyway. how secure is it? how featureful is it? how easy is it to write programs that use it correctly? how easy is it to make mistakes when writing programs for it?
02:55:34 <milesrout> geist: that's irc for you ;)
02:55:43 <Shockk> https://gist.github.com/shockkolate/e1563677b19ca6e3f62de0dfd924cb25
02:55:44 <bslsk05> gist.github.com: qsort.c · GitHub
02:55:48 <zid> It's because I am a terrible influence
02:55:52 <ekleog> Shockk: if it's in memory then the mov version is almost certainly going to be better
02:55:57 <Shockk> oh really?
02:56:07 <zid> Shockk: don't put qsort and performance into a sentence that doesn't also include "terrible"
02:56:07 <geist> my general feeling for how fast a modern x86 context switches is in it's probably in the low single digit usecs
02:56:18 <Shockk> zid: hahaha
02:56:22 <geist> as in, if you have a 4ghz machine, then that's 4000 cycles per microsecond
02:56:38 <johnjay> milesrout: sure that stuff matters too. but metrics do as well
02:56:41 <Shockk> I mean I basically also did the least efficient sorting method here too
02:56:42 <geist> so a full enter/exit, plys context switch is on the surface probably about a microsecond
02:56:53 <Shockk> so I'm not worried about the performance of this, just really curious
02:56:53 <zid> Your code is disgusting I hope you are aware of that
02:57:01 <geist> then of coruse you spread the cost of needing to refill the TLB over the next period of time, and that's a bit harder to measure
02:57:07 <ekleog> geist: fwiw, I had benchmarked on a cortex-m4, a context switch was ~5.3μs iirc
02:57:09 <geist> and that depends on how hot the page tables are in the caches
02:57:11 <zid> You can tell because of lines 12 through 16
02:57:19 <Shockk> I know that
02:57:22 <milesrout> johnjay: metrics is vague. tip: focus on actually getting something that does something before worrying about how competitive it is.
02:57:22 <zid> okay good
02:57:23 <Jmabsd> geist: wait wait, so for these "strapped context switches", about how many can you do in a second?
02:57:24 <Shockk> I don't like it either zid
02:57:24 <geist> ekleog: yah. depends on the clock cycles generally
02:57:32 <Shockk> just a cheap dirty impl
02:57:35 <geist> oh i dunno, 150k?
02:57:48 <ekleog> Shockk: xor-swap requires 3 memory accesses, mov-swap requires 2 memory accesses :)
02:57:55 <ekleog> (when you load from memory)
02:57:59 <zid> Shockk: y ou should probably learn about the break keyword >_<
02:58:00 <geist> per cpu, if you have a SMP machine that's sufficiently wide you can easily swap a million times a sec or more
02:58:02 <Jmabsd> geist: and if you optimize some more and then some more, with precalculated tables/structures, cutting FPU instructions if possible, and such?
02:58:08 <ekleog> errhm nvm I can't count
02:58:08 <geist> some.
02:58:19 <Jmabsd> also presume that it's fairly well optimized for being L1 memory accesses mostly?
02:58:30 <geist> but usually in the long run the actual low level cost of the save/load/exit is not the largest part of it
02:58:32 <Shockk> hmm would break really eliminate any of these nested blocks?
02:58:32 <milesrout> do you actually know that swapping is a performance problem for you?
02:58:36 <Jmabsd> geist: "wide" as in L1 cache size?
02:58:37 <milesrout> if not, stop thinking about it
02:58:40 <geist> usually that's fairly cheap compared to just the rest of the overhead of the operating system
02:59:00 <zid> Shockk: I don't understand the 'done' part
02:59:07 <geist> as in there are just so many instructions you need to run to maintain internal state, grab locks, look to see if a signal is delivered to this thread, etc etc
02:59:21 <Shockk> zid: that's called me being incompetent
02:59:24 <zid> oh okay
02:59:25 <geist> and once you optimize the low level context switch it ceases to be the 'long pole in the tent' so to speak
02:59:30 <Shockk> I forgot to set done to false if result > 0
02:59:38 <bcos_> Jmabsd: I don't remember seeing any articles; but if you understand paging (especially in long mode/AMD64) you'll know that for worst case one TLB miss is like 5 cache misses in series (e.g. about 1 thousand cycles), so depending on how many of those you actually trigger (which depends on the exact code and its access patterns, etc) you can see how that can get nasty
02:59:47 <zid> Shockk: okay, then you want instead, while(1) and if(result > 0) break;
02:59:48 <zid> not 'done'
02:59:57 <Shockk> ehhhh
03:00:20 <geist> *usually* the overhead of the syscall layer and validing all the args and doing stuff that makes you act like a kernel end up being where most of the time is spent. actually switching context on a modern machine (x86, arm64, power, etc) are pretty easy
03:00:27 <zid> very un-ehhh, it is definitely what you want, else your escape-path is super complicated
03:00:27 <Jmabsd> geist: interesting, so a million switches a second if you optimize hard. interesting.
03:00:38 <zid> you need to prefix every statement *after* result > 0 with if(!done && ..0
03:00:41 <milesrout> that's presumably just switching though right?
03:00:42 <zid> so that they don't get executed
03:00:50 <geist> i just made that up, but thats about what i've seen on a fairly modern, mid range cortex-i3 about 4ghz machine or so
03:00:59 <milesrout> as soon as you actually do anything on the things you're switching you're back down to far fewer presumably
03:01:06 <geist> that's the whole deal, like going in through a syscall and yielding
03:01:07 <Jmabsd> geist: (yes i see your point that other OS complexity takes over.)
03:01:13 <Shockk> zid: well no I mean I want everything after result > 0 to be executed
03:01:24 <zid> Shockk: Hmm, write it quickly for me?
03:01:27 <Shockk> I just also want it to set done to false, so that the loop continues
03:01:44 <Shockk> hang on
03:01:48 <Jmabsd> bcos_: what's the mechanism for that a TLB miss costs about 5 L1 cache misses, => 1 TLB miss = ~1000 cycles?
03:02:31 <geist> i may be off by up to a order or magnitude or so, may be somewhat faster, like nearly a million/sec
03:02:34 <geist> per cpu
03:02:35 <Jmabsd> bcos_: by TLB miss here you mean, roundtripping page fault handler, which is run when you access a page that you either don't have access to, or that is not mapped?
03:02:51 <bcos_> Jmabsd: CPU needs to fetch PML4 entry and misses all caches (~200 cycles), then needs to fetch PDPT entry and misses all caches, then needs to fetch PD entry, then PT entry, then the actual memory you wanted to access
03:03:04 <geist> so i'm looking at some benchmarks i ran the other day looking at the speed of the syscall interface on linux, for example
03:03:06 <Shockk> oh, no, I tried to figure out where to add the break, but no I can't just add a break zid
03:03:22 <zid> okay but please just fix it so I can look for real :P
03:03:22 <Shockk> that does the opposite of the logic I'm trying to achieve
03:03:24 <geist> on my threadripper, 4.1Ghz, modern AMD machine it nets just under 20 million syscalls/sec for the simplest possible syscall, getuid
03:03:30 <bcos_> Jmabsd: ..so something like "mov eax,[ebx]" can cost 1000 cycles (for a worst case)
03:03:33 <Jmabsd> bcos_: re:> What actually are your "memory privileges"? Is it just different attributes for pages, or segmentation in the same address space, or different virtual address spaces, or something else?
03:03:33 <Shockk> updated the gist
03:03:34 <zid> I have no idea where done needs to be fiddled
03:03:51 <geist> so that sets the bar right there, that's a full enter/exit from the kernel, so then divide that down by all the additional complexity of going in and yielding and context switching, and it's in the right ball part
03:03:56 <Shockk> the outer loop sets done to true at the start of each iteration
03:04:03 <Shockk> if it's still true at the end of the iteration, sorting is complete
03:04:07 <Jmabsd> bcos_: i think all activity in the whole system can share page set = virtual address spaces as you call it. what do you mean by "segmentation in the same address space"?
03:04:16 <Jmabsd> bcos_: i basically mean the R/W/X privileges to preexisting pages
03:04:25 <Shockk> so I set done to false if any of the member comparisons are > 0, i.e. lhs > rhs
03:04:37 <Shockk> because that means it needed to do at least 1 sort
03:04:38 <Jmabsd> bcos_: by "memory privileges" i mean, a piece of code's R/W/X access to given pages.
03:04:52 <zid> okay so for each element, indexed by 1, compare base to base+size (this is not legal is it?), if that result is greater than zero, do the swap of some kind, and if we did a swap, repeat the loop?
03:04:52 <geist> 20 mil/sec is about 200 cycles btw, for a syscall enter/exit
03:05:03 <bcos_> Jmabsd: Yeah - I figured that out since (most people don't call that a context switch - it's more like a "reconfigure all the page permissions without switching any context")
03:05:14 <zid> so it's basically a bubblesorty type thing?
03:05:32 <geist> so imagine it takes, say, 2000 cycles or so to do a full context switch between threads in the sam rprocess (that's now 2 million/sec) and then if it takes say 5000 cycles to switch between processes on the average
03:05:39 <geist> now you're just shy of a million/sec
03:05:51 <Shockk> hmm I mean I guess it just loops over each pair, comparing and swapping if needed, and looping back to the start if a sort was necessary
03:05:56 <Jmabsd> bcos_: you mean in worst case, the CPU's PML4 entry access means a cache miss, then PDPT entry access means a cache miss, and then PD and PT entry accesses mean cache misses too - so like 5-6 cache misses worst case
03:05:57 <Shockk> I don't remember what bubblesort is lol
03:06:06 <zid> Shockk: swap stuff until no swaps happen
03:06:13 <Shockk> oh okay, yes then I guess
03:06:18 <geist> so it's somewhere that ballpark, plus or minus an order or magnitude or so
03:06:30 <Jmabsd> bcos_,geist: if the particular PML4/PDPT/PD/PT in question are accessed frequently, i guess the likelihood of cache miss on their access is LOWER, that lowers the context switch cost then!
03:06:40 <zid> You could do it with continue and break, and not have 'done' but I'm not sure it'd be cleaner
03:06:40 <Shockk> bear in mind I didn't even test this code
03:06:49 <zid> so the swap case does continue; else break or whatever
03:06:54 <geist> yep. thats why it's very load dependent. there are terrible worst cases and okay best cases
03:06:57 <zid> Shockk: it doesn't compile as C, I can tell you that much
03:07:01 <zid> base + size is not legal
03:07:03 <Shockk> hmm really?
03:07:06 <geist> so you design for the best case and that sets the high water mark
03:07:08 <Shockk> it compiled for me
03:07:14 <zid> now add -ansi
03:07:18 <zid> or whatever
03:07:25 <geist> then assume in reality it runs somewhat worse than that, and you test and see what the deviation is
03:07:42 <geist> it's not precise, modern machines are hard to predict precisely, but you get a feel for it after a while
03:08:35 <Shockk> zid: oh, I get a lot more errors if I add -ansi
03:08:39 <zid> I bet :p
03:08:43 <geist> it's a complex series of tradeoffs, each with different weights. ie, it might be awful to have to dump the TLB, but then the cpu is pretty good at refilling it
03:08:46 <Shockk> fails to understand what `restrict` means
03:08:54 <zid> oh right, bah
03:08:57 <zid> try -std=c99
03:09:09 <geist> one of the things that folks get attached to too much in osdev is they get fixated on a particular thing and try to optimize that to the detriment of everything else
03:09:10 <zid> gcc compiles this to 'ret' for me
03:09:26 <zid> Not sure if that's because of a logic error, or the UB
03:09:27 <Shockk> hmm what
03:09:27 <geist> when in reality there are lots of compromises and optimizations, and you can't do all of them, and each of them has a different weight
03:10:03 <Shockk> it definitely compiles to something tangible for me
03:10:32 <zid> oh it's the bool crap I think
03:10:41 <Shockk> oops
03:10:43 <geist> that's some bool crap!
03:10:52 <geist> stepped in what? bool crap!
03:10:53 <Shockk> boolshit
03:11:20 <geist> Jmabsd: anyway, that what you were looking for?
03:11:35 <zid> Why is gcc not exploding at base + size :/
03:11:37 <bcos_> Jmabsd: Let's look at this from a different perspective (optimising what you've got). You've got 2 major performance problems - the cost of rewriting all the RWX flags, then the cost of TLB misses after. To avoid the need to rewrite all the RWX flags you could just use a different PML4(so that each context is already has all the RWX flags set up). To avoid the TLB misses you can use the PCID feature. 2 major performance problems disappear
03:11:58 <Shockk> I thought ptr + integer was fine to do
03:12:18 <zid> void has no size
03:12:29 <zid> so how much + should +integer do, 1*integer? 4*integer, etc
03:12:32 <geist> and gcc will generally let you add to anyway, as an extension
03:12:41 <zid> yea but I did -std=c90 so it shouldn't let me
03:12:51 <Shockk> hmm let me look up the standard
03:13:07 <jp> void will be size one as an extension (functions are also size 1 funnily enough)
03:13:12 <jp> but per standard they have no siE
03:13:17 <jp> size*
03:13:20 <geist> normally ptr + intger it adds the size of the eelemtn you're pointing to
03:13:21 <jp> stupid phone
03:13:31 <zid> Looks like you need -pedantic, okay
03:13:33 <Jmabsd> bcos_: if i have precalculated all the PML4 structures, down to what level of expense does that bring my security context switches?
03:13:38 <geist> and adding to a void * is not allowed, except gcc lets you do it, and it treats it like a char
03:13:45 <zid> -pedantic produces the warning, -W -Wall -Wextra etc do not
03:13:52 <geist> right
03:14:11 <jp> i dont remember how big bool is to gcc
03:14:16 <jp> might also be 1
03:14:26 <zid> geist: That's because of history I think, void * is newer than 'C' and used to just be a typedef for char * until it got better semantics or something
03:14:31 <zid> I'm not up on my 1985 compiler tech
03:14:32 <bcos_> Jmabsd: With PCID; if the working set (all TLBs from everything being switched to/from) fits in TLB; it reduces security context switch costs to "almost zero"
03:14:36 <Jmabsd> bcos_: what's the point of Intel's 4096-elements Process Context IDentifier feature, how does it help security context switches? anyhow i'd have a lot more than 4096 contexts, but, maybe in one given second or 3 seconds etc., 4096 could suffice
03:14:36 <zid> but I remember reading something like that
03:14:40 <jp> zid: convenience
03:14:43 <jp> nothing more
03:15:04 <geist> prolly so
03:15:19 <Jmabsd> bcos_: interesting
03:15:26 <zid> There was a lot of existing K&R C with void * == char * I believe
03:15:29 <jp> people doing arith on void * want char scaling 99% of the time
03:15:31 <zid> and they wanted to support it
03:15:34 <jp> saves a cast is all
03:15:39 <geist> Jmabsd: also note that many other architectures have it, it's not a particularly new thing, intel just added it for x86 in a not so useful way
03:16:07 <Jmabsd> geist: PCID not a new thing, interesting.
03:16:20 <bcos_> Jmabsd: Without PCID; when you change virtual address spaces (change CR3/change PML4) the CPU automatically flushes all of the TLB entries (that weren't specially marked as "global"); which causes TLB misses. With PCID the CPU puts a "which virtual address space this TLB entry belongs to" into the TLB entry itself, so that it doesn't need to flush any TLB entries and you don't get the TLB miss costs
03:16:26 <geist> it's not a new concept in computer architecture, it's just new to x86 (showed up around nehalem and then got extended in haswell)
03:16:44 <geist> note that AMD cpus still dont support PCID
03:16:52 <Jmabsd> geist,bcos_: first, thank you very much for having taken a minute to help me navigate this problem domain. so PCID is the CPU maintains 4096 different memory access contexts internally and allows you to do superfast switching between. that's useful.
03:17:08 <Jmabsd> at least that allows you to use the PCID set as a ""cache"" of 4096 security contexts
03:17:19 <geist> correct
03:17:19 <Jmabsd> as an optimization over worst case :)
03:17:32 <bcos_> Ironically; I think AMD was first to add the basic concept to 80x86, but they only did it for virtual machines and not normal processes (which was incredibly short sighted..)
03:17:33 <geist> which is precisely what linux does. it somewhat dynamically assigns the PCID to the active processes
03:17:40 <geist> bcos_: yah
03:18:04 <geist> bcos_: fairly certain that's where it came from anyway, since they already had to add the bits for the L2 TLB to track the vmid
03:18:26 <geist> that's precisely how it works on ARM: sizeof(ASID) == sizeof(VMID) because it just reuses the same bits in the TLB
03:18:33 <Jmabsd> bcos_: interesting. initially was PCID somehow unsuitable for use to switch between processes?
03:18:52 <geist> kind of. nehalem's version of it was very limited so not a lot of systems picked it up
03:19:08 <geist> haswell added some additional context flushing instructions that made it marginally useful
03:19:17 <Shockk> hmmmm
03:19:22 <geist> and then meltdown fixes made it essentially mandatory (on intel cpus)
03:19:29 <geist> so lots of systems crammed it in in the last year
03:20:15 <geist> it ends up being one of those complexity vs hard-to-measure performance gain, so wasn't traditionally that big of a deal
03:20:21 <Shockk> can zid or someone else verify something for me from n1570 (C11)?
03:20:44 <geist> sure, dumping the TLB is expensive, but then so is a lot of stuff, and adding a bunch of complexity to the code path to support the feature is not a slam dunk
03:21:31 <geist> but with meltdown, now you have to swap mmu context on *every syscall* so it becomes almost mandatory to mitigate that cost
03:21:34 <zid> Shockk: no, because you forgot to ask it
03:21:40 <Shockk> hang on lol I'm typing
03:22:00 <Shockk> I see in section 6.5.6 that either both operands shall have arithmetic type, or one operand shall be a pointer to a complete object type and the other shall have integer type
03:22:11 <Shockk> so it's a question of whether or not void* is a complete object type
03:22:13 <zid> void * is not a pointer to a complete object type, naturally
03:22:16 <zid> void is not an object
03:22:20 <zid> it's specifically not one
03:22:28 <Shockk> I understand that, but hang on
03:22:32 <zid> else void *a; *a would be legal
03:22:42 <Shockk> 6.2.5 says void is an incomplete type and cannot be completed
03:22:56 <Jmabsd> bcos_,geist: so, if my TLB entries / PML4/PDPT/PD/PT are all in the L1 cache, how much time for a memory security context switch are we talking? also, having the **same** memory addressing for ALL activites should make switches cheaper shouldn't it? (i.e. instead of having separate virtual memory configurations = set of page number => physical memory page mappings, all activities have the same and memory access security context
03:22:57 <Jmabsd> switches only regard pages' RWX configurations)
03:23:12 <Shockk> but clause 20 of that section, a bit further down the page, says "A pointer type is a complete object type"
03:23:30 <Shockk> and it doesn't say it conditionally, so I'm wondering if I'm reading this right
03:23:54 <Mutabah> Shockk: A _pointer type_ is complete
03:24:03 <zid> 6.2.5.20?
03:24:03 <Mutabah> e.g. `void*` is complete, but `void*` is not
03:24:21 <Mutabah> So, `void**` is a pointer to a complete object, but `void*` is not a pointer to a complete object
03:24:27 <bcos_> Jmabsd: To change the RWX for one page you need to write to the page table entry and then invalidate the TLB (so that CPU doesn't keep using a stale TLB entry with the same old RWX). That alone is going to cost you 12 bucketloads of cycles (and significantly worse if it's multi-CPU)
03:24:31 <zid> not 6.5.2.20
03:24:45 <Shockk> zid: it's 6.2.5, clause 20
03:24:52 <Shockk> also OH, I realized my mistake
03:24:59 <zid> Any number of
03:24:59 <zid> derived types
03:24:59 <zid> can be constructed from the object and function types,
03:25:03 <zid> That's what .20 is for me
03:25:04 <Shockk> "shall be a pointer to a complete object type"
03:25:18 <Shockk> so while the pointer is a complete type, it doesn't point to one
03:25:20 <zid> sorry for the shitty paste I hate pdf pasting :/
03:25:21 <geist> Jmabsd: i wouldn't call it a bucketload. i can't get you hard numbers
03:25:24 <Shockk> also I love the pdf pasting
03:25:24 <Shockk> :D
03:25:38 <geist> i honestly dont know, it's just something you try to minimize
03:25:41 <zid> D E R I V E D T Y P E S
03:25:49 <geist> but if you gotta do it then it happens, and it's intel/amds job to make it as cheap as possible
03:25:53 <zid> but yea, 6.2.5#20 is that
03:25:59 <zid> not whatever you said
03:26:04 <Shockk> hmm weird
03:26:17 <Shockk> zid: oh no, wait, I mean that is the correct section
03:26:21 <Shockk> just I don't have numbers for the -
03:26:35 <Shockk> it's the subclause for "A pointer type ..."
03:26:38 <zid> it's talking about derived types :P
03:27:16 <zid> void is not an object type, as mentioned, so that section is not talking about void * being a derived type
03:27:47 <Jmabsd> bcos_: > write to the page table entry and then invalidate the TLB (so that CPU doesn't keep using a stale TLB entry with the same old RWX). That alone is going to cost you 12 bucketloads of cycles
03:28:22 <Jmabsd> bcos_: how much is "12 bucketloads" here? the operation regards local-core usage only, the cores operate independently. "multi-CPU" as you mention here shouldn't affect anything i guess
03:28:35 <Shockk> zid: no no, it's an object type, but not a complete one, "it is an incomplete object type that cannot be completed"; but like I said, I realized my mistake,
03:28:42 <bcos_> Jmabsd: How much space are you talking about reconfiguring?
03:28:49 <Shockk> as 6.5.6 says the thing being pointed to (i.e. void) must be complete
03:28:59 <Jmabsd> bcos_: first, so, the PCID feature mitigates this totally?
03:29:30 <Shockk> hm I guess I need to fix my code then
03:29:42 <zid> also reading the C11 spec is weird
03:29:47 <zid> C90 spec best spec
03:29:49 <Jmabsd> bcos_: amount of space.. hm... if it's lazy work then ~~~500KB of data = ~~~125 pages. however could go as high as ~~~100MB = ~25,000 pages
03:30:05 <bcos_> Jmabsd: Using "1 PML4 per module" avoids the need to modify RWX flags totally. PCID avoids the cost of TLB misses after
03:31:19 <bcos_> Jmabsd: OK, so 25000 pages with some overhead a write and an INVLPG? For a random unknown CPU that might 25000*20 = 500000 cycles
03:31:21 <Jmabsd> bcos_: as long as i have 1 PML4 per module, what's the Memory Access Privileges Context (= MAPC :) ) Switch cost roughly approx? and.. with PML4+PCID?
03:31:57 <klange> the Hobby OS Museum demands more sacrifices^W donations https://i.imgur.com/dNocXda.png
03:32:34 <Jmabsd> bcos_: and 125 pages would take how much? so, the 500,000 cycles figure is.. for when you don't have the PML4 precalculated and not use PCID, yes?
03:32:57 <Jmabsd> and 100% cache misses? :-}
03:34:32 <bcos_> Jmabsd: 500000 cycles is modifying 25000 page table entries and doing INVLPG 25000 times, with no cache misses at all (and not counting TLB miss costs anywhere).
03:35:35 <bcos_> Jmabsd: ..and 125 pages would be more like 2500 cycles maybe (still with random unknown CPU, no cache misses, no TLB misses)
03:36:17 <Jmabsd> bcos_: aha. on an <24 months old not-cheapest Intel CPU, how much would 25,000 and 125 pages be?
03:36:53 <bcos_> Jmabsd: How large is your L1 cache? 25000 page table entries won't fit in a typical 64 KiB L1 cache..
03:39:23 <Jmabsd> bcos_: aha yes i see your point, so, as soon as you do *ANY* data work outside the L1 cache, data processing costs skyrocket anyhow.
03:40:03 <Jmabsd> bcos_: now, if the memory privileges for the 25,000 pages were *precalculated* in PML4/via PCID, then the MAPC-switch would be way way faste4r yes?
03:40:38 <Jmabsd> bcos_: in what situation do you need to run the INVLPG instruction >>>1 times?
03:42:12 <bcos_> Jmabsd: Yes, would be faster. INVLPG only does a single page, so if you change the RWX for 10 pages you "need" to do INVLPG 10 times
03:43:35 <bcos_> ("need" because... in theory you can do lazy TLB invalidation when you increase access and not when you decrease access, but in practice that doesn't help unless its single-CPU because you get page faults that cost more than INVLPG)
03:44:13 <bcos_> Sigh
03:45:14 <bcos_> klange: Almost had a heart attack. Opened your screen shot earlier and didn't really look, then opened KCalc and did something, then forget what I was did. Looked at your screenshot and Kcalc was in the middle - thought you ported KCalc to your OS for a few seconds.. ;-)
03:46:14 <klange> my OS isn't even in this screenshot :D
03:46:39 <Jmabsd> bcos_: when a normal Unix/Windows context-switches processes, it is *NOT* pumping INVLPG instructions though is it?? instead it has PML4 and maybe PCID, and that *relieves* the OS of the need of INVLPG pumping?
03:46:43 <bcos_> klange: No? I recognised the status bar at the top..
03:46:51 <klange> That's gnome-shell.
03:47:29 <bcos_> D'oh
03:47:31 <klange> ToaruOS's panel was inspired by some early gnome-shell design mockups, though it doesn't really look a lot like the current incarnation of gnome-shell if you compare side by side.
03:48:22 <bcos_> Jmabsd: Normal Unix/Windows uses "different PML4 for each process" and doesn't do INVLPG when switching between processes
03:49:00 <klange> There's a 9front, Shrine (TempleOS w/ networking), Aquila, Dilos (SunOS derivative), TravorOS, Tyndur (which I definitely selected "English" for, but still get all the messages and text in German), Vanadium, Sortix, PowerNex, and the gigantic window in the background is Glidix selecting the largest resolution QEMU would give it.
03:49:05 <bcos_> Jmabsd: PCID doesn't releive the need for INVLPG, it avoids the TLB misses after
03:49:25 <johnjay> so is virtualbox a more mature VM system than Qemu?
03:49:32 <johnjay> even though Qemu supports gdb?
03:49:51 <klange> "more mature" is hard to measure
03:50:05 <bcos_> johnjay: I'd say VirtualBox is more user-friendly - nice GUI thingy to configure/manage virtual machines
03:50:21 <johnjay> i will say the qemu documentation is dog poop, as useful as it is
03:50:23 <bcos_> ..but apart from that I think they stole most of Qemu'\s code anyway
03:50:28 <johnjay> the vbox documentation is very extensive
03:51:04 <klange> QEMU is older, supports more architectures, has a much better software emulator, and is fundamental to KVM usage in production environments.
03:51:30 <klange> VirtualBox has corporate maintenance, is more user friendly, supports hardware virtualization on other platforms than Linux, and has better EFI integration.
03:52:29 <bcos_> johnjay: For OS development/testing purposes, my advice is "use all the things!" - you can't test on too many different (real or virtual) machines
03:52:59 <klange> The great thing about both of them is they are FOSS so there is zero cost beyond disk space to using both of them.
03:57:19 <zid> neither of them are bochs though
03:58:49 <shikhin> bcohs
03:59:05 <zid> also none of them are bcos
03:59:19 <klange> bcosh
04:01:15 <klange> a bochs is fine also https://i.imgur.com/VE1ef8u.png
04:01:43 <zid> getting bochs to build is getting harder and harder though :P
04:01:44 <mischief> tyndur is weird.
04:01:48 <klange> note: i do not recommend running a software compositor in bochs
04:01:49 <zid> I have several edits to its makefile now
04:02:07 <klange> I have had no trouble building bochs, but I think we had this discussion previously?
04:02:26 <zid> it doesn't link against pthreads correctly and fails with global -fPIE and other stuff
04:03:11 <klange> i even have an old bochs build for mainline toaru
04:03:24 <mischief> vanadium looks neat
04:04:00 <klange> yeah, piotr's got a nice GUI going there with a bunch of apps, feels very much like the classic late 90s OSes :)
04:11:31 <johnjay> bochs is... interesting
04:11:43 <johnjay> i took one look at the macro-infested source and ran screaming away into the night
04:23:25 <klange> boch is a decade older than qemu, it still very much has that 90s code smell to it
04:26:41 <Jmabsd> bcos_: how much time does modern Unixes/Windowses spend doing INVLPG:s?
04:27:22 <Jmabsd> bcos_: ah yes that was what you said - PML4 for each process means no need for INVLPG:s, yes?
04:28:10 <bcos_> Jmabsd: You still need to do them when you change a virtual address space (e.g. someone calls "mmap()" or "sbrk()" or whatever)
04:28:14 <Jmabsd> bcos_,geist: here then, if you're switching from 'process' A which has its own PML4 to 'process' B which has its own PML4, and the process is *only* the kind of RWX privilege switching I talked about, then, how much time does it take?
04:28:47 <Jmabsd> bcos_: if I map an 1TB file, do I need to do INVLPG once for each 4KB page of the whole 1TB interval? :o
04:30:20 <bcos_> Jmabsd: Normally if you map a 1 TB file the OS marks the pages as "not present" (which involves nothing if they're already not present); but then when you access a page it fetches the data from disk and maps a page and makes it "present"
04:31:24 <bcos_> ..but in this case there's a loophole - most CPUs don't store "not present" in the TLB so you don't need to INVLPG for most CPUs
04:33:09 <geist> Jmabsd: when switching between processes like that it takes the same amount of time, dumping the tlb is fast
04:33:28 <geist> it's the cost spread across over the next N seconds as the second processes touches one or more pages
04:33:54 <geist> dumping the TLB is fast, it's the fact that as a result of dumping it newer page accesses will potentially take extra time
04:34:11 <geist> but then amount of extra time new acceses takes varies somewhat between nothing at all and some
04:34:25 <geist> and the some is hard to quantify, and how much between nothing at all and some is hard to quantify
04:34:33 <geist> except that probably on the average it takes some
04:35:08 <geist> it's just another level of 'the cpu has a cache and the ability for it to cache and/or predict your behavior is what makes it fast'
04:35:29 <Jmabsd> geist: i totally see what you mean. yes. in a way this is the inherent complexity of running software.
04:35:34 <geist> so when you dump a part of the cache you inhibit its ability to hide memory access latency, so it's hard to quantify it
04:35:57 <geist> also remember it may not be slower at all, even after dumping the TLB
04:36:10 <Jmabsd> geist: here (relating to memory privileges), what's going on is the CPU is the code execution environment which needs to know some things in order to be able to approve or deny memory accesses - it's symmetrical to program code
04:36:28 <geist> because the cpu is speculatively loading stuff like crazy as it runs code, and it may be able to completely hide the latency of the TLB miss by speculatively loading it ahead of time
04:36:38 <Jmabsd> geist: The whole TLB is represented by the PML4 structure?
04:36:54 <geist> the TLB isn't, the data structure that is referenced when it gets a TLB miss is
04:37:16 <geist> but for the most part a TLB entry contains all of the information it got from the page table entry when i has a TLB miss
04:38:26 <geist> so you can think of the TLB as caching entries out of the page tables, or maybe you can think of the page tables as being a secondary cache to the TLB itself
05:08:49 <Jmabsd> geist,bcos_: do all modern CPU 64bit architectures use the "ring" concept for segregating code that has direct access to hardware IO facilities versus code that not?
05:09:30 <johnjay> this guy is saying his qemu hangs with black screen when trying to mount a win10 cd
05:09:38 <johnjay> is it possible to use gdb to diagnose that?
05:10:07 <Mutabah> With the gdb stub, you can at least see what instruction is running...
05:10:14 <Mutabah> Maybe get an idea what it might be doing
05:10:24 <Mutabah> but without source, it's gonna be hard to figure out exactly what
05:10:39 <johnjay> that's what i thought. i keep thinking os development is all about reading assembly code
05:10:42 <johnjay> but i guess that's not quite the case
05:11:27 <tyler569> if you have a concept of what it should be doing, reading assebly can be useful. Reading it from first principles to debug or reverse engineer is much mroe difficult
05:12:00 <geist> Jmabsd: kinda. generally speaking it's priviledged or not
05:12:19 <geist> but most modern hardware is completely memory mapped, so the paging system acts as a fundamental gating mechanism
05:12:29 <johnjay> tyler569: ah ok. along the same lines i was wondering if you could observe intel cpu behavior with just the qemu debugger
05:12:36 <geist> so that means the kernel can decide if a user space process gets access to registers or not
05:12:44 <johnjay> but that probably involves writing the barebones kernel
05:12:45 <geist> or at least some entity can, and the paging system acts as the gatekeeper
05:13:23 <tyler569> johnjay: you can to some extent, but you don't gain a ton more insight with qemu gdb than you do just running gdb on the host
05:13:44 <tyler569> honestly the qemu console is more interesting, you can query all kinds of interesting things with the `info` commands
05:13:50 <johnjay> tyler569: ah ok. i was thinking along the lines of rings and the first startup things the processor does
05:14:37 <tyler569> oh, I can't say I've tried to step through pre-bootloader. I know it does some amount of it since it does a bunch of SMM resets at start time
05:14:40 <geist> yah you're not going to learna whole lot by trying to gdb debug it, since you dont have source
05:14:49 <geist> but you can also just use the qemu console to stop the emulation and see what the cpus are doing
05:14:59 <geist> which will tell you about as much as its useful to know, which isn't a lot
05:15:11 <geist> as in, is it in protected mode? did it make it to 64bit? etc
05:15:19 <geist> but after that it'll just be addresses that aren't helpful
05:17:20 <cpplearner> Guys, my professor once said that, the linux does not return from the system call upon timer interrupts (probably ignoring it), unless its job is done. Is it really the case?
05:17:40 <mischief> why wouldn't it be
05:18:29 <johnjay> tyler569: un fucking believable. the qemu documentation mentions the qemu monitor but not how to actually start it. -_-
05:18:54 <klange> It's always running.
05:18:59 <klange> It's not something you start...
05:19:10 <klange> And in the newer interfaces it's accessed from a menu.
05:20:46 <klange> What docs are you reading?
05:26:29 <geist> johnjay: OH MY GOD. ITS A TRAVESTY
05:26:39 <johnjay> https://qemu.weilnetz.de/doc/qemu-doc.html
05:26:41 <bslsk05> qemu.weilnetz.de: QEMU version 3.0.0 User Documentation
05:27:16 <johnjay> geist: i'm annoyed because i specifically skimmed that page before trying to remember if qemu had that exact thing
05:27:53 <geist> qemu docs are all over the place
05:28:14 <geist> anyway, yeah you can get to the console any number of ways, sometimes it's a menu that flips the window over
05:28:31 <geist> or you can redirect it to stdio or a tcp port or pipe or whatnot
05:28:36 <johnjay> geist: maybe qemu 3 has a menu, idk. i'm using qemu2 from msys and there is 0 menu items
05:28:57 <geist> indeed. it also depends on how it's built. and if you're running the msys version then yolo
05:29:03 <klange> you are using qemu... in windows? for what purpose?
05:29:07 <geist> that's the garbage bin of programming environments
05:31:37 <mischief> lipstick on a pig.
05:32:01 <klange> Oh apparently qemu can do hardware acceleration on Windows since 2.9.0? I did not know this.
05:32:10 <geist> klange: not so bad then!
05:32:16 <geist> dunno about the msys version though
05:33:55 <klange> This article I am reading also indicates the Windows version of QEMU has menus, including the View menu that I believe has the Monitor option.
05:34:07 <mischief> maybe windows malware will now just be qemu.exe with a disk image appended
05:34:35 <mischief> or if they built with libcurl, builtin http url to boot from :))
05:34:40 <klange> You kinda have to jump through hoops to set up HAXM.
05:35:29 <klange> Also, given this is HAXM, it presumably only works with VT-x, so Intel-only.
05:36:27 <johnjay> actually i have qemu on windows and on msys2 in windows and also on linux in virtualbox in windows
05:37:42 <johnjay> ah i see yes so the windows qemu does have a menu option
05:37:51 <johnjay> you have to hit ctrl-alt-G to get out of the window to click on it...
05:37:59 <Jmabsd> geist: when you have IO "privilege", can you write to *any* hardware, or there's some other OS-managed structure that determines addresses? I guess DMA memory access security simply falls under virtual memory management, doesn't it? so then there's IRQ/interrupts and a bit more [i'm not well read on how to interface PCI]
05:38:28 <Jmabsd> geist: there is some PCI device access that _not_ is done via memory addresses nor IRQ calls, isn't it, or?
05:38:36 <klange> ? https://www.qemu.org/screenshots/qemu-debian-wheezy-shell-with-haxm.png
05:40:11 <Jmabsd> geist: jumping to another question - actually having all OS processes share memory mappings works fine *except* for mmap() which consumes so much space, right? (because the ~~~64bits of virtual addressing space couldn't address more than ~~~64bits of physical memory anyhow, so the virtual memory can be used just to put memory access privilege annotations on the ram)
05:52:08 <bcos_> Jmabsd: I think you need to divide your time more equally - e.g. 8 hours sleeping; then 8 hours reading stuff from manuals, tutorials, etc; then 8 hours asking geist questions non-stop
05:52:19 <bcos_> ;-)
05:53:23 <zid> I think him and geist should just get married already
05:54:58 <Jmabsd> lolol.
05:55:27 <johnjay> is geist an expert?
05:55:29 <Jmabsd> i was baking these questions for a couple of years, sorry if i'm overwhelming or anything =)
05:55:36 <johnjay> (on irc all usernames look alike)
05:56:18 <Jmabsd> bcos_: =) meanwhile, how does PCI interaction work, as suggested in my Q here? =)
05:56:53 <zid> He's an expert at feeding vampires
05:57:44 <bcos_> Jmabsd: I like to think of PCI as a kind of network where devices send packets of stuff at each other
05:57:57 <johnjay> haxm is something to speed up android emulation... sounds interesting
05:58:02 <zid> bcos_ also likes to think about shaved squirrels though
05:58:10 <zid> so don't get too interested in what he likes to think about
05:58:15 <Jmabsd> bcos_: packet reception is signalled by bus/CPU to software by *ONE* interrupt call, yes?
05:58:35 <Jmabsd> "interrupt storm" is when software or drivers malbehave so the bus/CPU eats the CPU by pumping interrupt cals
05:59:59 <bcos_> Jmabsd: Some packets (from device to CPU) are interrupts, some (from CPU to device) are reads/writes to registers or memory mapped IO, some (from device to system memory) are bus mastering/DMA, some are errors when things go wrong, ..
06:00:47 <Jmabsd> bcos_: from the software, facilities are used except for normal memory accesses (=DMA) and IRQ interrupt handlers?
06:00:54 <bcos_> ..but there's also stuff between the device and CPU and memory (PCI bridges, IOMMU maybe, etc) that can convert/translate
06:02:04 <bcos_> DMA = the device talks to memory without the CPU being involved != memory accesses from CPU
06:02:05 <Jmabsd> right, so you have IOMMU configuration
06:03:35 <Jmabsd> bcos_: but normally you configure hardware simply by writing to memory addresses, and waiting for the hardware to mutate memory addresses? (yes I see DMA just means "bridging of devices with designated areas of y our RAM")
06:05:06 <Jmabsd> bcos_: ah, re mmapping, what about the Q to geist > "actually having all OS processes share memory mappings works fine *except* for mmap() which consumes so much space, right?"
06:05:17 <bcos_> Often you configure/interact with devices by reading/writing to addresses (but not always - e.g. maybe there's IO ports)
06:06:36 <bcos_> Having all processes share which memory mappings how and how often?
06:08:44 <Jmabsd> bcos_: the "IO ports" facilities, where can i read about them? those are low-bandwidth more peripheral applications though, aren't they?
06:09:28 <Jmabsd> bcos_: "Single address space operating system" - yes that model =)
06:09:57 <bcos_> "IO ports" is mostly like a special extra address space, with special instructions to access it
06:10:10 <Jmabsd> bcos_: all processes sharing memory mappings would help processes to allow other processes to access their memory, no?
06:10:53 <Jmabsd> bcos_: interesting, so virtual memory has become the sole interface today whereby you interface hardware, that's interesting.
06:11:14 <bcos_> Jmabsd: All processes sharing memory would help malicious processes access everything whether they're allowed to or not
06:11:38 <bcos_> ..but apart from helping malware, there's no advantage
06:12:21 <Jmabsd> bcos_: but the malicious process would need to have R/W/X access to said memory, right - when that is _not_ the case, malicious processes would _not_ be helped, right?
06:12:55 <Jmabsd> bcos_: if you're in a place where processes are deeply intertwined, having access to each other's memory and being able to do "process-shared memory mutexes" and what not would be useful wouldn't it
06:12:58 <bcos_> (note that with "each process has its own virtual address space" you can map the same memory into 2 or more virtual address spaces so that the memory is shared; but that usually involves a special agreement/arrangement and isn't default)
06:13:46 <Jmabsd> bcos_: sure. as relevant, the OS could appoint _the same_ (virtual) memory addresse-s/ranges in both processes too, so it looks like exactly the same thing.
06:14:53 <bcos_> Jmabsd: As we talked about earlier, to improve performance of "changing RWX" you need to give each process its own virtual address space anyway. Unfortunately (with various security vulnerabilities - e.g. Spectre) RWX alone is not secure enough
06:16:37 <bcos_> Jmabsd: The OS could appoint _the same_ (virtual) memory addresses/ranges in both processes; but there's no reason to do that (it just makes things more restrictive/limited for no reason)
06:17:12 <Jmabsd> bcos_: re spectre, the conclusion that if a given code should not access some memory anyhow then don't map it at all - interesting. aha.
06:17:28 <Jmabsd> bcos_: this principle applies on all CPU architecftures, right - AMD64, ARM64, Sparc, PPC64, what not
06:17:31 <Jmabsd> MIPS64
06:17:52 <Jmabsd> with respect to memory mapping and device IO they're fairly similar, aren't they
06:18:07 <bcos_> Jmabsd: The other thing is that for most "64-bit" CPUs the virtual address space is actually smaller (e.g. 48-bit) and there are processes/software that needs more (and Intel is planning to extend it so that a single process can use more)\
06:18:42 <Jmabsd> bcos_: re 48 bits yes i know. sometimes for ridiculous reasons, I think mmap() has a 256GB limit on ARM64 Linux
06:20:04 <bcos_> Realistically; you could split "48-bit virtual address" into a 16-bit process ID followed by a 32-bit address within the process's area; and end up with an OS that has the same limitations as 32-bit CPUs had a decade ago
06:20:09 <bcos_> ..or
06:20:52 <Jmabsd> bcos_: wait, if I am *not* using PCID, *but I am using* separate PML4 structures, then how much will a memory access privileges context switch cost approooox?
06:21:38 <Jmabsd> so in this case there will be TLB misses that will have a cost.. actually why
06:22:10 <geist> did you get your IO stuff dealt with before getting onto this 48 bit stuff?
06:22:16 <bcos_> ..you could assume that modern CPUs are single address space, where you have the same "Process ID + address within the process' area" but it's actually a 12-bit process ID (from the PCID feature) plus a 48-bit virtual address = 60-bit
06:22:48 <geist> i honestly wouldn't tell them that
06:22:54 <geist> that generally confuses em more than anything else
06:23:05 <geist> yes you can think of it that way, but it's merely a conceptual model
06:23:33 <geist> since that's not what it 'looks like' from a programming point of view to regular code
06:25:34 <zid> segment selectors are back they just don't want you to know!
06:25:39 * zid passes out tinfoil hats
06:26:00 * klys declines
06:27:04 <geist> because i'm a masochist i just booked another train trip down to california in a few weeks
06:27:15 <geist> instead of flying
06:27:25 <zid> I'm assuming american trains are made of hobo skin and prayers?
06:27:37 <geist> pretty much, and steak dinners and lots of old people
06:28:54 <bcos_> Wouldn't train be less restrictive? More space/less cramped, able to bring liquids, able to use laptops, ..
06:29:15 <zid> lose a week of you rlife..
06:29:20 <geist> basically yes
06:29:37 <geist> i kinda like getting stuck on it for a day, you have to talk to people, can read book, do work
06:29:43 <geist> and look out the window and see mountains n stuff
06:29:50 <geist> they're not fast at all
06:30:16 <geist> i took one a few years ago from seattle to washington DC. took 4 days
06:30:39 <graphitemaster> geist, You had a wagon wheel break but were able to replace it from supplies. Oh you have dysentery now.
06:30:47 <klys> I live near a passenger train station...
06:30:56 <geist> yah and sometimes you have to eat some of the other passengers
06:31:08 <graphitemaster> now you have cholera
06:31:17 <graphitemaster> you've managed to break 3 arms too somehow
06:31:27 <geist> actually one of the routes goes over Donner's pass
06:32:26 * bcos_ has no idea how far it is to California
06:32:50 <graphitemaster> bcos_, better buy a lot of food and wagon wheels, you're going to need 'em
06:33:05 * bcos_ would need scuba gear
06:33:14 <graphitemaster> I don't know that version of the game
06:34:05 <klys> distance | from | Sydney, New South Wales, Australia to | San Francisco, California, United States = 11932 km (kilometers)
06:34:18 <geist> lets see. about 4400 km from seattle to washington dc
06:34:40 <geist> about 1300km from sea to SF
06:35:10 <geist> it's about a 13h drive if you dont stop
06:35:17 <geist> and 24h if you take the train
06:36:52 <graphitemaster> and about 2 if you fly
06:37:44 <geist> yah
06:37:53 <geist> usually about 1:50
06:38:17 <geist> i'd actually like to take the trans canadian at some point too
06:38:30 <geist> it's hella expensive, but the trip through the rockies is apparently phenomenal
06:38:35 <johnjay> trains are full of spies and espionage like in movies right
06:39:20 <geist> and i looked into the trans-australian one too, there's one that runs from sydney down to adelaide or thereabouts
06:39:31 <geist> and then another leg that goes to perth
06:39:49 <geist> johnjay: yeah, absolutely
06:40:41 <bcos_> Wait... Who is the idiot that put Washington DC on the other side of the country to Washington?
06:41:34 <geist> note that sea -> washington dc is a little bit longer than madrid to moscow
06:41:55 <geist> which is about the same distance as sydney to perth
06:42:03 <geist> 4400 vs 4100 vs 3900 km
06:42:58 <geist> bcos_: same unorginal bastards that always put new in front of some thing in europe
06:43:10 <geist> new amsterdam, new york, new hampshire, etc
06:43:31 <geist> at least most of the crap on the west coast is named after some saint or is a native american name
06:43:44 <_mjg> is there a way to annotate a function as typically succeeding?
06:43:44 <geist> except for mountains, which white folk loved to rename after some white guy
06:43:52 <_mjg> (returnin g0 or whatever)
06:44:15 <geist> _mjg: good question, that would be nice
06:44:16 <_mjg> the goal is to not have to spread likely/unlikely at each consumer
06:44:19 <_mjg> yea
06:44:30 <geist> i *suppose* you could wrap it in an inline function with a likely clause maybe?
06:44:44 <_mjg> if i had just one, then ye
06:46:24 <bcos_> _mjg: A function that typically succeeds to do what?
06:46:32 <_mjg> whatever
06:46:47 <_mjg> i indicate the typical return value (e.g. not-null, 0 or whatever else)
06:46:53 <geist> i guess you really want to basically say 'this generally returns a particular value'
06:47:01 <_mjg> and then consumers checking the return value get automagic prediction
06:47:08 <_mjg> simple exmaple would be malloc
06:47:18 <_mjg> as typically returning non-null
06:47:32 <_mjg> another one would be fd -> file lookup
06:47:49 <_mjg> which basically never fails unless you are screwing around
06:47:50 <zid> superlikely(true, malloc);
06:47:55 <zid> you just need to be able to design that macro :P
06:48:00 <zid> glhf
06:49:18 <_mjg> thanks dawg
06:49:21 <geist> even if you did put a wrapper around it
06:49:39 <geist> and use __builtin_expect, dunno precisely what that would look like
06:49:54 <geist> if (__builtin_expect(foo(), 0)) ...
06:49:57 <geist> return 0?
06:50:56 <_mjg> probably #define crap(x) { int _err = _crap(x); if (likely(err == 0)) return 0; return err; }
06:51:09 <_mjg> with ( ) and without reutrns, but you get the idea
06:51:33 <zid> what arch are those hints actually worth more than the instruction decode bw or whatever?
06:51:52 <burzos> _mjg: BSD has __predict_false, __predict_true; for example https://github.com/NetBSD/src/blob/64b8a48e1288eb3902ed73113d157af50b2ec596/sys/sys/ktrace.h#L327
06:51:54 <bslsk05> github.com: src/ktrace.h at 64b8a48e1288eb3902ed73113d157af50b2ec596 · NetBSD/src · GitHub
06:51:56 <geist> most. in general it's not that it adds a hint at the instruction level, since most dont have that (POWER/PPC does, i remembe)
06:52:17 <_mjg> x86 has static hints but they are not recommended
06:52:23 <zid> oh it's used more like pgo?
06:52:25 <geist> but in general it causes the compiler to structure the code so that it assumes the unlikely path is a branch to some code at the end of the function
06:52:26 <_mjg> the real win stems from generating code
06:52:36 <zid> right
06:52:37 <_mjg> the unlikely case is moved out of the way and requires a jump forward
06:52:51 <_mjg> which is statically predicted as not taken
06:52:53 <geist> basically assuming the branch predictor is cold then the usual static rules are forward conditional branches are assumed to be not taken
06:53:06 <geist> and i think that's fairly universal
06:53:29 <_mjg> basically you can turn a jumping-jack workout and turn it into a walk in the park
06:53:43 <geist> it is lovely to look at the dissassembly of a function like that and see this nice straightforward hot path
06:53:45 <_mjg> if certain cases are almost always true
06:53:51 <_mjg> geist: fuckin' a
06:54:02 <_mjg> right now i'm looking at the opposite
06:54:19 <_mjg> clang decided that fget is likely to fail and added func prologue at the very beginning
06:54:26 <_mjg> and doing actual work requires jumping over it
06:54:27 <geist> wonder if you can safely build a macro around it
06:54:30 <_mjg> :(
06:54:40 <geist> that just basically wraps the function with likely(...., 0)
06:54:59 <_mjg> burzos: i know, that's likely/unlikely and does not take care of the problem
06:54:59 <geist> as in does the compiler care if it see s abuiltin_expect by itself outside of a conditional statement
06:55:22 <_mjg> burzos: with them you have to do things by hand, e.g. foo = maloc(..); if (__predict_false(foo == NULL)) ...
06:55:29 <geist> my experience is that clang doesnt' seem to honor likely/unlikely nearly as much as gcc does
06:55:38 <_mjg> that's m yexperience as well
06:55:45 <zid> clang thinks it knows better ;)
06:55:51 <geist> pretty much
06:55:59 <burzos> Yeah annotating the function def is much nicer.
06:56:12 <zid> although I've read a couple of writeups on example optimization paths of pieces of code
06:56:18 <geist> note there are arches that *do* have a likely/unlikely thing. PPC comes to mind. iirc there's a '.' that you can put at the end of an instruction that is a static predictor
06:56:26 <geist> but i think all modern machiens wouldn't do that because it's a waste of a bit
06:56:27 <zid> and it seems.. rather frenetic, and that it would be easy to accidentally discard the hints
06:56:33 <olsner> my hunch is that a lot of well-intending coders put those annotations where it would not help / make things worse
06:57:05 <zid> some path turning a mov into a lea and accidenally not the lea with a dot after it, or whatever
06:57:14 <geist> perhaps. i've generally only seen likely/unlikely in kernel code
06:57:16 <_mjg> olsner: that used to be the case in linux
06:57:31 <geist> most other coders i've seen dont know it exists
06:57:33 <_mjg> people were way too willing to plop it
06:57:39 <geist> and/or are writing tricky ass c++ that makes it hard to use
06:57:49 <_mjg> however, there are pretty blatant cases
06:57:57 <_mjg> like that malloc example
06:58:16 <geist> my guess is it's much easier to make hot code slightly better with it than it is to mess up non hot code
06:58:35 <geist> as in you can basically randomly toss it down in most plain ass unimportant code and it'd make very little difference, but a few hot places it may help a lot
06:58:39 <geist> but that's just a hunch
06:59:06 <_mjg> as you noted gcc takes hints more seriously
06:59:11 <zid> it definitionally doesn't matter in non-hot paths
06:59:14 <_mjg> and i trust gcc folks to do better job :>
06:59:43 <_mjg> the bsds are slower single-threaded in part because of the compiler
06:59:55 <_mjg> for instance there is no asm goto equivalent
07:00:06 <_mjg> linux uses it to nop-out dynamic probe points
07:00:22 <zid> https://godbolt.org/z/IEkxN6 We should all switch to msvc its codegen is flawless
07:00:23 <bslsk05> godbolt.org: Compiler Explorer
07:00:26 <_mjg> fbsd has to put conditoinal jumps in there
07:00:55 <olsner> maybe likely/unlikely could be useful as an indication of what the developer thought would happen (e.g. this is the hot path, that is the cold path)
07:01:32 <geist> i've actually used it in non hot/cold path stuff but more like 'hey this is 50/50 but if it's true, then i'd like this path to be very pretty'
07:01:33 <geist> and that's a bit hard to explain in code review or whatnot
07:02:14 <geist> like one path can be very nicely optimized, and the other one is going to be a shit-show no matter what, so may as well hint it to make a nice job out of the good path
07:02:20 <_mjg> it is not recommended to plop in hints unless there is a high skew in one direction
07:03:04 <geist> what'st he rationale there? the compiler will probably figure out what i was thinking anyway?
07:03:21 <geist> (i'm perfectly willing to change my ways, just curious)
07:03:58 <_mjg> there are instructions like cmov
07:04:21 <_mjg> which will do pretty good job much cheaper than a jump would if you miss
07:05:07 * geist nods
07:05:31 <geist> but the hint will probably force it to go all out and always do a cmp/jmp?
07:05:31 <_mjg> basically the compiler can genreate code which is slower than it could be for both cases, but with 50/50 execution the total perf is better
07:05:44 <geist> okay, i think you got me sold
07:06:09 <_mjg> with prediction you tell it to pessimize the other case
07:06:14 <geist> i'd buy that for a dollar!
07:06:15 <_mjg> of course clang does not always listen
07:06:17 <_mjg> :>
07:06:30 <geist> yah thats all based on the idea that the compiler actually cares about what you told it
07:06:34 <geist> and isn't a little insolent child
07:06:41 <_mjg> gcc mostly is not
07:06:44 <geist> clang is like the Millenial of compilers
07:06:53 <_mjg> in general i found clang fucks you up if there are loops involved
07:07:12 <zid> oh I h adn't considering it actually fucking the cold case to make the hot case faster
07:07:17 <_mjg> but if the code is straight top bottom it is way more likely ot listen to the hint
07:07:18 <zid> so yea, hints could definitely be destructive int hat case
07:07:42 <zid> but by definition neither case is aaactually a hot path.. soooo..
07:07:59 <olsner> istr cmov is not as good as you might think - e.g. because cmov has both code (value?) paths as inputs it ends up forcing work that could otherwise be discarded after resolving the branch
07:08:17 <zid> I've never seen gcc generate a cmov
07:08:27 <geist> i think i have, from time to time
07:08:43 <geist> but maybe only in the case wher eyou're doing straight up a cmov in code and it's such a perfect fit
07:09:13 <zid> I convert all my ternaries into multiplication by 1 or 0 so the compiler gets confused
07:09:32 <geist> surprised it actually gets confused by that
07:09:35 * bcos_ slaps zid - use 0 and -1 so you can AND
07:09:47 <zid> I wouldn't know I was memeing
07:10:58 <_mjg> olsner: cmov perf depends on the microarch, it is quite dcent now
07:11:16 <_mjg> i don't remember gcc, but clang will plop cmov
07:12:12 <olsner> ah, good to know, instruction perf knowledge is a bit of a perishable
07:12:22 <_mjg> see instructoi ntables by agner fog
07:12:37 <zid> ^
07:13:18 <geist> it's useful enough construct that ARM64 actually added cmov and 0/1 register set based on conditional instructions
07:13:22 <zid> 2 2 x x x 2 1 *scrolls up for the column names
07:13:26 <_mjg> so i just grepped my linux kernel, there are few cmovs
07:13:33 <geist> something like cmov dest, A, B, condition
07:13:42 <zid> it can operate them basically at full speed apparently
07:13:52 <johnjay> _mjg: grep works for hex characters or?
07:13:58 <_mjg> objdump -d
07:14:02 <johnjay> ok
07:14:13 <_mjg> so i'm grepping freebsd now, fuckton of results
07:14:23 <_mjg> clang likes its cmov
07:15:18 * zid figures out how to objdump a bzimage
07:17:35 <zid> hmm I need to find the gzip header in a vmlinuz I think.
07:17:48 <zid> no idea how to grep in hex
07:17:48 <_mjg> get yousrelf a debug image
07:17:53 <zid> od spits out on multiple lines and stuff
07:17:59 <zid> _mjg: oh can I just do make debug or something
07:18:16 <_mjg> well if you compile your own kernel, then you have a vmlinuz file which is just an elf
07:18:24 <zid> it is not just an elf
07:18:27 <_mjg> otherwise get a debug package t oget the file
07:18:33 <_mjg> it's not the bzimage
07:18:34 <zid> it's a binary blob that makeimage.sh or whatever turns into a bzimage
07:18:40 <_mjg> bzimage is separate
07:18:56 <zid> vmlinux != vmlinuz
07:19:09 <zid> vmlinux.bin: data
07:19:25 * zid ponders what setup.elf is
07:19:29 <_mjg> there should be a vmlinux file in the main dir
07:19:36 <_mjg> it's what i meant (typos)
07:19:45 <_mjg> once you copmile the kernel
07:19:47 <zid> oh wow, there totally is
07:19:50 <zid> how did I never notice that
07:19:57 <zid> I did my find on arch/
07:19:57 <_mjg> ye, so that's the elf
07:20:27 <zid> I have.. approximately 10478 cmovs apparently
07:20:32 <zid> my cpu is pretty good at them says agner
07:21:01 <zid> and 76410 lea's
07:21:04 <zid> so a 76 lea to cmov ratio
07:23:24 <geist> ah yes, the all important lea to cmov ratio
07:24:00 <zid> Might as well peg it to some common opcode so get a sense of scale
07:24:03 <zid> I figured
07:24:21 <zid> ret may have been better
07:26:57 <zid> anyone got anything older than a sandy bridge willing to do the same?
07:27:04 <geist> probably mov maybe
07:27:12 <geist> it's the glue of instructions
07:27:13 <zid> mov is a bad grep though :P
07:27:21 <_mjg> :-P
07:27:26 <_mjg> grep .
07:27:28 <geist> behold a site i heard of today
07:27:33 <zid> You'd at least get combined cmov and mov
07:27:44 <zid> and movs and rep mov and crap
07:27:48 <geist> prepare for your mind to be blown:
07:27:51 <geist> https://regex101.com/
07:27:51 <bslsk05> regex101.com: Online regex tester and debugger: PHP, PCRE, Python, Golang and JavaScript
07:27:52 <zid> is it googles
07:28:18 <zid> Like I want to run a regex on 100MB of objdump output and trust it to be correct, *I* wrote the regex
07:28:28 <geist> yes. yes you do
07:28:39 <geist> you do!
07:28:42 <zid> I had a problem, I needed to count movs, I now have two problems
07:28:47 <johnjay> HOLY SHIT I ALWAYS WANTED THIS THANKS
07:28:54 <zid> (I've used that site before actually)
07:28:59 <_mjg> wow, nice
07:29:00 <zid> I was dumping combat logs in to make a parser
07:29:12 <zid> my capture groups were being fucky trying to eyeball it
07:29:13 <geist> i didn't know about it until today, but i think it'll go next to godbolt as one of those things that i should give them money for
07:29:35 <geist> (but probably wont)
07:30:05 <zid> saving it up to give to me, good idea
07:30:10 <zid> it IS my birthday next month afterall
07:30:25 <_mjg> geist: that's the website i did not know i inneded
07:30:56 <geist> yeah i might actually use regexp more often now
07:31:06 <geist> i usually dread it
07:31:15 <zid> .*but ((s|)he|the .*|one of the .*) (just|deftly|easily|) parries the blow.*
07:31:16 <_mjg> although pcre (php) is a little bit icky, like running java on linux in a former solaris shop
07:31:16 <zid> :D
07:31:19 <_mjg> :>
07:32:08 <_mjg> (did you know there are people running java on smartos (illumos, former solaris) thanks to linux emulation)?
07:32:15 <_mjg> that's kind of perverse
07:32:30 <olsner> aww, it doesn't have any of the flavors of regexps I use
07:33:50 <zid> It supports the kind I write
07:33:52 <zid> regexb
07:33:55 <zid> regex, but they're bad
07:33:56 <mischief> wat
07:34:11 <mischief> .theo
07:34:11 <glenda> I love this conversation.
07:34:13 <zid> You can tell because I wrote them
07:34:22 <johnjay> that's the odd thing about regex, is it seems everyone has their own version. pcre, grep re, emacs re, etc
07:34:41 <FireFly> heh, I helped with some stuff on regex101 once upno a time
07:34:45 <FireFly> it's a nifty project
07:34:49 <zid> an FireFly
07:35:02 <klange> well, regex is just extensions to traditional regular expressions
07:35:07 <FireFly> a zid
07:35:18 <zid> any relation to the firefly who wrote the rt2560 driver I have laying around?
07:35:18 <klange> and everyone's got their own ideas of what extensions should exist and how they should work
07:35:25 <FireFly> zid: nope
07:35:29 <zid> shame
07:35:52 <FireFly> nor to the person reversing Pictochat on the DS, even though they also camelcase their nick, and it's very adjacent to the kinda stuff I enjoy doing
07:36:44 <zid> FireFly: That is that firefly :P
07:36:57 <zid> the rt2560 driver is to comm. with a DS
07:36:59 <FireFly> aha
07:37:07 <FireFly> that makes sense, heh
07:37:26 <FireFly> I've reversed other things in relation to nintendo consoles, DS included, but not that :p
07:37:28 <zid> FireFly: Have you seen the Jet Li movie "The One"?
07:37:31 <FireFly> nope
07:37:37 <FireFly> should I?
07:37:37 <zid> Shame, you are unaware of your destiny then
07:37:41 <FireFly> :o
07:37:48 <FireFly> I guess I should
07:38:11 <zid> I was about spoil it but if you're just going to watch it I uess not :P
07:38:38 <FireFly> no spoilers!
07:40:03 <johnjay> FireFly: that's so cool senpai notice me plz
07:40:12 <FireFly> hi johnjay
07:40:32 <FireFly> I don't do anything terribly cool though
07:40:33 <zid> FireFly: I only badly and slowly reverse a silly PC tetris game, nintendo stuff sounds more fun
07:40:51 <zid> well apart from that one time someone asked me to RE a gamecube game and it turned out just to have full debugging symbols in the elf
07:41:30 <zid> hmm.. fun nintendo RE projects.. alttp/oot randomizer?
07:43:16 <johnjay> FireFly: i was hanging out in dolphin channel but I keep expecting something to come up about reversing what some ppc assembly does or something
07:43:23 <johnjay> but it seems that rarely matters
07:43:44 <zid> I'd imagine the dynarec core gets very little attention
07:43:49 <johnjay> my thinking was if you could understand the ppc assembly of a gc game you could debug the program better
07:43:51 <zid> especially on that level
07:44:05 <zid> The supershaders thing was a great read :D
07:44:43 <FireFly> dolphin internals stuffs is always really fascinating to read about, and the monthly reports are always ace
07:44:56 <zid> I keep meaning to get around to writing a crappy dynarec, I've only ever done interps
07:45:15 <zid> I figured once my gb emulator gets into a good state I'd just do it there, but I never got there
07:45:18 <FireFly> my reversing projects are weird in that I don't really reverse executables much at all, and mostly prefer to stare at hexdumps of data and try to figure out what's going on
07:45:24 <FireFly> but hey, sometimes that's useful too
07:45:45 <zid> as in, image formats etc?
07:45:58 <johnjay> FireFly: the report thing is kind of creepy though. like... who is that directed at? a report makes it sound like something you give to your boss
07:46:00 <FireFly> Sure, or archive formats, or bytecode formats, or compression
07:46:04 <FireFly> or sound formats
07:46:12 <zid> I've got a couple of odd file formats to do for the tetris game but it turns out they use so few of the bytes in the headers it's sort of pointless
07:46:15 <FireFly> basically the kinda stuff that game devs seem to enjoy reinventing
07:46:18 <zid> they're all the same 'class' within that ormat
07:46:34 <zid> Might as well just hardcode all that crap and seek past it
07:46:50 <FireFly> aww but understanding what it means and how it could be altered is more fun
07:46:52 <johnjay> FireFly: that sounds awesome! And senpai has in fact noticed me!
07:47:09 <zid> FireFly: right but like I said, everything is the same class, so there are no other code paths :(
07:47:17 <zid> What the game software hilariously does is just compare the first 256 bytes of the file with a copy of the 'stock' header and bails if it isn't right
07:47:32 <FireFly> ah
07:47:35 <zid> so documenting what it probably means is sort of pointless
07:47:41 <FireFly> I'd still do it tbh :p
07:47:45 <johnjay> what i am curious now is what tools you guys use for all that stuff
07:47:52 <FireFly> a hexdump tool, vim
07:47:52 <johnjay> windbg? hex editor?
07:47:52 <zid> IDA and a hex editor
07:47:55 <johnjay> objdump -d?
07:48:03 <johnjay> oh ok, i haven't used IDA much
07:48:15 <zid> cheat engine for finding shit, windbg for debugging shit, IDA for analyzing shit, hex editor for staring at shit
07:48:27 <FireFly> I never really got into IDA/r2/etc, and hex editors.. idk, I used to use okteta a bit, but I don't really feel like it's useful much either
07:48:28 <zid> hrrrmmm some program just made a window for like a quarter of a second then closed it and I have no idea what it was...
07:48:35 <johnjay> as expected of a hardcore reverser like FireFly! He only needs vim to grasp The Source!
07:48:36 <FireFly> I prefer to just annotate a text hexdump
07:48:44 <FireFly> no I find it easier :p
07:48:58 <zid> FireFly: This thing has like 4MB of auto-generated .c I've mostly cleaned up and made workable at this point
07:49:01 <FireFly> colours help a ton when trying to find patterns
07:49:20 <FireFly> https://raw.githubusercontent.com/FireyFly/hexd/master/meta/screenshot.png <- like that
07:49:23 <zid> I can read C at 1000mph, even nicely coloured assembly I can't :P
07:49:34 <zid> FireFly: wanna see what I made my hex editor do?
07:49:37 <FireFly> much easier to spot patterns in binary data
07:49:42 <FireFly> zid: yeah!
07:49:48 <zid> I need to find an elf..
07:50:05 <_mjg> most compiler generated assembly generates few simple patterns
07:50:17 <_mjg> and few tricks
07:50:33 <zid> aww I lost that file
07:50:35 <_mjg> s/generates/consists of/
07:50:59 <_mjg> problmes start when someone tries to crank out the last cycle out of the microarch
07:52:48 <zid> anyway this is the wrong struct but you'll get the idea
07:52:56 <zid> https://cdn.discordapp.com/attachments/417023075348119556/492241728456228865/unknown.png
07:53:09 <johnjay> how do you annotate a text file?
07:53:11 <zid> I made full ELF structs so I could debug my elf linker script
07:53:23 <zid> and made my hex editor overlay them onto ELF files dynamically
07:53:42 <FireFly> cute
07:53:48 <FireFly> That seems to be a popular feature in hex editors these days
07:53:54 <zid> It's suuuper useful
07:53:59 <FireFly> okteta has a struct-y thingy too
07:54:01 <klange> neat
07:54:18 <johnjay> when i googled HexD I got Firefly's github
07:54:21 <johnjay> Senpai has a github!
07:54:36 <FireFly> I've been meaning to check out kantai-struct, for "defining a struct and matching against file"
07:55:06 <zid> https://cdn.discordapp.com/attachments/417023075348119556/492242267688402944/unknown.png IDA <3
07:55:55 <johnjay> FireFly: what's the ranges of color?
07:56:01 <johnjay> just the grey, white, purple?
07:56:22 <zid> https://cdn.discordapp.com/attachments/417023075348119556/492242559410634752/unknown.png better than staring at this and squinting for 4 hours
07:56:35 <klange> FireFly: oh i remember when you were writing pixd
07:56:53 <FireFly> johnjay: 0x00, "low" control bytes, printable ASCII, "eight bit set", and 0xFF
07:57:13 <johnjay> right that's what i guessed.
07:57:18 <FireFly> you can map them to whatever colour you want, but the default scheme (and the one I use) is black (or very dark grey), green, white, purple, red
07:57:28 <FireFly> klange: oh yeah
07:57:34 <FireFly> pixd was fun
07:57:40 <zid> What's a pixd
07:57:43 <FireFly> and people seem to actually find it useful somehow
07:57:51 <FireFly> it's like hexd but blockier
07:57:59 <FireFly> https://github.com/FIreyFly/pixd
07:58:00 <bslsk05> FireyFly/pixd - 🔍 Colourful visualization tool for binary files (18 forks/358 watchers)
07:58:12 <zid> Whatever this is making windows and closing them or making my cusor flash to busy is driving me nuts
07:58:16 <zid> do I a malware or something
07:58:27 <johnjay> FireFly: that is pretty genius though when you think about it
07:58:29 <FireFly> johnjay: re. annotating text file, http://xen.firefly.nu/up/Z2Sound.baa.note.html is the best example I could find offhand
07:58:32 <bslsk05> xen.firefly.nu: Z2Sound.baa.note
07:58:57 <johnjay> I had an idea for debugger interface, you know when you step in a debugger like ollydbg or whatever and values change?
07:59:02 <zenix_2k2> one thing, idk where to ask this but sometime i encounter a really weird defect of my computer, my entire laptop freezes and i couldn't move my cursor, so is it because of my RAMs, my GPUs or my GPU's drivers ???
07:59:03 <johnjay> and you think, well what did that value change from?
07:59:14 <johnjay> i was thinking you could either make it so you click on it and it shows you the previous value
07:59:27 <johnjay> OR it shows a small animation of a the previous number floating away
07:59:33 <johnjay> like in World of Warcraft
07:59:38 <zid> my data visualizer is broken
07:59:39 <klange> zenix_2k2: go ask a support forum for your OS, this is a channel for writing OSes from scratch
08:00:23 <johnjay> like a small superscript
08:00:37 <zid> there we go
08:00:49 <zid> https://cdn.discordapp.com/attachments/417023075348119556/492243693999357972/unknown.png my hex editor has a super ghetto version of that
08:01:18 <zenix_2k2> klange: ok then
08:01:27 <johnjay> FireFly: how does the pixd work, is it meant to just colorize it like in the picture?
08:01:30 <geist> could be anything
08:02:16 <johnjay> FireFly: what would be nice is like use warm colors for data and cool colors for instructions or something
08:02:18 <geist> never thought about it, but how much code goes into implement a regexp parser?
08:02:32 <zid> a lot if you want it to be fast
08:03:54 <johnjay> zid: you created your own c-like language??
08:03:56 <FireFly> johnjay: it just maps bytes to colours, you can provide your own colour map, but the default one maps the high nibble to hue and the low nibble to brightness
08:04:19 <zid> johnjay: what?
08:04:19 <geist> i always assumed that a purely interpreted straightforward slow implementation is probably a few thousand lines of tight code
08:04:47 <johnjay> zid: https://ziglang.org/
08:04:47 <bslsk05> ziglang.org: The Zig Programming Language
08:05:13 <zid> is that a really shitty troll or can you not read
08:05:21 <FireFly> geist: I'd say a straightforward interpreted version is at most like.. 1-2k
08:05:30 * geist nods
08:05:46 <johnjay> zid: i was guessing zid = zig
08:05:52 <FireFly> I wrote one at one point (but only for the CS kind of regular expression)
08:06:18 <geist> johnjay: take off every
08:06:28 <klange> zxzx... hm, seems I'm missing a format specifier...
08:06:53 <klange> zx... what is z supposed to mean...
08:07:13 <johnjay> FireFly: I was thinking in terms of more structural map, like in a c program color the main() red, the functions blue, the data section a diff color palette.
08:07:22 <geist> zxzxzx
08:07:24 <klange> ah, size specifier for size_t
08:07:26 <geist> zyzzy even
08:07:32 <johnjay> or like using colored borders, that might work better
08:07:59 <[REDACTED]> johnjay: no, that's scientes
08:08:05 <[REDACTED]> who isn't even in this channel
08:08:20 <johnjay> ok
08:08:43 <klange> ugh FireFly why you use err/errx/warn/warnx those aren't standard
08:08:50 <FireFly> johnjay: I mean, you could do that, sure. I wanted to see if this was a useful way to visualize arbitrary binary data (and not just executables)
08:08:56 <[REDACTED]> klange: because everything implements them
08:08:58 <klange> https://i.imgur.com/4iyUDwo.png
08:09:04 <[REDACTED]> they're not hard to implement
08:09:07 <FireFly> klange: they're handy, also I think it was suggested by others that I use them :p
08:09:15 <[REDACTED]> (i.e. me)
08:09:16 <klys> s⌇zz⌇e
08:09:23 <klange> stop giving me work to do
08:09:25 <[REDACTED]> no
08:09:36 <FireFly> oh hey my thing runs in your thing
08:09:37 <FireFly> woo
08:09:47 <johnjay> FireFly: yeah the idea of outlines would work really well, like it would show you the relative size of the .data and .text sections of the executable. or just different chunks in a jpeg or something
08:10:00 <FireFly> Sure, but that requires domain-specific knowledge :p
08:10:11 <FireFly> and if you already know it's a jpeg, you can just process it as a jpeg
08:10:29 <johnjay> oh so you mean like, identify if it's a jpeg or a ppc or a text file
08:10:49 <FireFly> sure
08:11:09 <FireFly> or more generally than that, because if it's a standard format `file(1)` would pick it up of course
08:11:26 <FireFly> "is this encrypted/looks-like-noise, or does it have some structure to it?"
08:11:37 <FireFly> "are there any areas of this file that stand out? where should I start looking?"
08:12:13 <johnjay> ah ok. i thought it was more specific like, how many data chunks and how big are they in this jpeg file or something
08:12:20 <FireFly> nah
08:12:37 <FireFly> oh and it's also very handy for finding like, indexed bitmaps in old games
08:12:51 <johnjay> so how does it work on compressed zip files or finding compressed files?
08:13:19 <johnjay> yeah i'm trying to understand the C source example. lots of short green or yellow lines a few pixels long
08:13:44 <FireFly> it literally just shows you the data inside the file without processing it, so if it's a zip, it won't decompress or extract it or anything
08:13:56 <klange> lots of green = ascii text
08:14:04 <FireFly> the C source code is all printable ASCII, except for the line breaks
08:14:27 <FireFly> hold on
08:16:34 <FireFly> https://twitter.com/FireyFly/status/867015829212340228 was the original twitter thread
08:16:34 <bslsk05> twitter: <FireyFly> Toying with visualizing binary data with colours. https://pbs.twimg.com/media/DAhB9tFXkAEcOTp.png [Example output for 8 file formats: C source code, tar, PPC and ARM machine code, mp3, webm, OTF and TTF.]
08:16:41 <FireFly> it explains the palette and has some more examples
08:17:51 <Jmabsd> geist: > <geist> did you get your IO stuff dealt with before getting onto this 48 bit stuff?
08:18:21 <Jmabsd> geist: kiinda - what i understand is IO is mostly done via memory addresses (and IRQ calls). including the whole "other IO" category.
08:18:26 <FireFly> johnjay: it's like.. FILE *f = fopen("the file", "r"); while (!feof(f)) { int ch = fgetc(f); printf("\x1B[48;5;%dm \x1B[m", palette[ch]); }
08:18:27 <FireFly> in essence
08:18:48 <FireFly> just using the raw bytes in the file to pick colours from palette, is all
08:18:49 <Jmabsd> bcos_: <bcos_> ..you could assume that modern CPUs are single address space, where you have the same "Process ID + address within the process' area" but it's actually a 12-bit process ID (from the PCID feature) plus a 48-bit virtual address = 60-bit
08:18:52 <Jmabsd> bcos_: what do you mean?
08:21:56 <johnjay> FireFly: what's this arcan you tried that segfaults?
08:22:19 <johnjay> are those vt100 escape sequences??
08:22:23 <FireFly> some other visualization tool
08:22:47 <FireFly> johnjay: well, it's SGR, but with the 256-colour extension, yes
08:23:06 <FireFly> most terminal emulators these days do at least 256-colour, often also 24-bit colour
08:23:15 <klange> 24-bit's nice
08:23:19 <FireFly> agree
08:25:19 <bcos_> johnjay: The point I was trying to make there is that the advantages most people think they're going to get from SASOS (with all the other disadvantages) are the same advantages that every OS gets anyway (without all the disadvantages); because it's mostly the same if you have a more flexible definition of "address"
08:27:05 <johnjay> FireFly: this is a really neat concept! but i can't think of a clever way to extend it
08:27:24 <johnjay> maybe color based on 4-byte or 2-byte chunks?
08:27:39 <FireFly> Well, the actual tool uses the top block glyph, to fit two "pixels" into one terminal character cell
08:27:41 <johnjay> or allow that to be set as an option
08:27:52 <FireFly> johnjay: hmm i guess
08:27:55 <zid> humans are really bad at comparing hues or luminocity across 2D space
08:28:07 <zid> we do okay at comparing dark green to light green, but medium red to medium blue is basically impossible
08:28:28 <zid> so trying to do it on 2 axes at the same time, which you'd need to to represent more than 8 bits at a time is basically not going to work
08:28:32 <FireFly> I'm not sure it's practical to have 64k or 4M lookup tables though
08:28:52 <FireFly> er 4G*
08:29:13 <FireFly> would probably want to describe the mapping in another way in that case
08:29:36 <zid> I recommend electric shocks
08:29:48 <zid> 16 bit value, lower 8 is voltage, upper 8 is pulse duration
08:29:58 <FireFly> johnjay: honestly I don't use pixd much, but it was a fun experiment
08:30:03 <FireFly> I mostly just use hexd..
08:30:13 <FireFly> others seem to find use for pixd though
08:30:16 <johnjay> the only other thing i can think of is to map the 'delta' - i.e. for each byte compute (byte value) - (byte value - 1) and then colorize that.
08:30:26 <johnjay> ok fair enough
08:31:29 <johnjay> so you'd get a sense of how the bytes are changing as the file goes on
08:31:53 <FireFly> mmm
08:32:39 <FireFly> I'd like to try to write a file(1)-like utility that reiles more on statistics like that (well, correlation between pairs of bytes, e.g.) to finegrprint "formats" (which might not strictly be one format, but more a class of related formats) at some point
08:32:54 <johnjay> i.e. text portions would be very low since the ascii chars are close together, then binary data would look different since it have more variety i guess
08:32:58 <FireFly> I think that'd be more useful in some cases than fingerprinting by magic number
08:33:24 <bcos_> Jmabsd: Ooops - my last message was supposed to be for you (tab completion fail!)
08:33:31 <johnjay> file uses specific bytes?
08:33:33 <FireFly> like "this file contains a lot of probably Z80 machine code" is useful even if it's not exactly a known binary container
08:33:52 <johnjay> i thought file already did that. maybe i was mistaken?
08:33:53 <FireFly> sure, AFAIK file just relies on fingerprinting by magic numbers
08:33:58 <johnjay> eh
08:34:01 <johnjay> that's stupid
08:34:05 <FireFly> not really
08:34:20 <johnjay> well if you just want that then sure
08:34:20 * zid remembers the open office 'file' bug that was funny
08:34:21 <FireFly> it's very handy :p
08:34:28 <johnjay> but i'd rather have the fuzzy version for every day stuff
08:35:15 <zid> https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161/comments/28
08:35:19 <bslsk05> bugs.launchpad.net: Comment #28 : Bug #255161 : Bugs : cupsys package : Ubuntu
08:36:05 <FireFly> zid: hahaha
08:36:09 <FireFly> that's an amusing bug
08:36:22 <zid> Only on tuesdays :)
08:36:36 <johnjay> FireFly: this is so weird, i could have sworn there was already a program that did fuzzy matching as you describe
08:36:49 <Jmabsd> bcos_: there can be partial SASOS. there can be actually not mapping intervals, but making them appear out of nothing when they're first accessed. yeah i see your point that SASOS is not a silver bullet or anything like that.
08:36:58 <Jmabsd> also SASOS makes mmap() superproblematic!!
08:37:49 <FireFly> johnjay: there's binwalk, which doesn't do fuzzy matching (or, well, it might do as part of its things, but the primary thing it does is walk through the file and look for magic numbers indicating starts of various formats at any offset)
08:38:28 <FireFly> so you can easily find e.g. a jpeg inside an HDD image, or the compressed data inside a compressed archive
08:40:11 <johnjay> FireFly: check out the heuristic that sublime text editor uses to search, it's out of this world
08:40:13 <johnjay> https://blog.forrestthewoods.com/reverse-engineering-sublime-text-s-fuzzy-match-4cffeed33fdb
08:40:15 <bslsk05> blog.forrestthewoods.com: Reverse Engineering Sublime Text’s Fuzzy Match – Dev Curious
08:40:41 <johnjay> FireFly: that is mind blowing to me that a fuzzy matcher like that doesn't already exist. what. the. frack.
08:41:32 <FireFly> Hm, I'll check out that post later, but thanks for the link!
08:41:43 <johnjay> so i assume it would have a 'score' for each type, like a gif score, a tar score, a ppc assembly score
08:41:53 <johnjay> then it just displays the highest % scores for the file?
08:42:23 <FireFly> eh, there's a lot of ways you could approach it
08:42:24 <johnjay> e.g. 98% chance jpeg file, 80% chance C file, 50% chance random noise, etc?
08:43:01 <FireFly> and I'm not an exepert on statistics :p
08:43:07 <FireFly> but something like that, probably, yeah
08:43:44 <johnjay> hrm, i wonder if you would need fancy statistics for something like that
08:44:11 <johnjay> or even better... machine learning
08:44:17 <johnjay> then you train it with the files you want it to recognize (!)
08:45:07 <johnjay> i just mention that because of a so-called "alien" chess program I read about recently
08:45:18 <johnjay> it trains by playing against itself thousands of times
08:45:37 <johnjay> so the result is it plays really bizarre styles that no human has ever seen before
08:47:18 <johnjay> so you could train it with the standard 'file' program or something
08:47:38 <zid> Then it's shit, honestly.
08:47:52 <zid> e4 is best by test, not best by people ignoring g2
08:47:54 <zid> err g3
08:48:14 <zid> there are plenty of neural net chess engines, none play weird
08:48:18 <zid> and they have 3800 elo and stuff
08:52:52 <FireFly> johnjay: you could train a classifier, sure, not certain it's necessarily the best approach
08:52:58 <FireFly> might be though
09:06:23 <oo_miguel> hmm can i use c functions (as from newlib) as fread/fwrite on raw filedescriptors or do I need a "FILE *" for that as returned by fopen?
09:08:42 <zid> oh shit gcc 8 supports naked for x86 apparentl
09:13:32 <zid> https://gcc.gnu.org/ml/gcc-patches/2017-07/msg01968.html
09:13:34 <bslsk05> gcc.gnu.org: Uros Bizjak - [PATCH, i386]: Implement attribute ((naked))
09:13:50 <oo_miguel> ok froget what I asked
11:10:24 <Shockk> just a quick curious question; what exactly should getppid be expected to return if there is no parent process? i.e. what if it's called from the root of the process tree?
11:13:04 <zid> that would be init
11:13:26 <zid> I'd say "whatever the fuck, who cares, init shouldn't need to care" but maybe there's a reason that isn't true
11:13:32 <Shockk> hmm okay
11:14:10 <Shockk> maybe just 0 or its own pid then
11:15:33 <zid> 0 on linux apparently
11:15:42 <zid> grep "Ppid" /proc/1/status
11:15:48 <Shockk> huh okay
11:15:58 <klange> "If the caller's parent is in a different PID namespace (see pid_namespaces(7)), getppid() returns 0."
11:16:05 <klange> which is a situation that likely includes "doesn't exist"
11:17:12 <Shockk> klange: that's linux-specific though, I was wondering if there's a more general expected behaviour in POSIX but it seems not, POSIX just specifies that getppid is always successful
11:17:47 <Shockk> 0 seems to make sense though, more than its parent being itself
11:17:56 <zid> shrug, nothing and everything makes sense
11:17:59 <bcos_> What's supposed to happen if the parent terminated (and maybe the same PID got recycled) before getppid() is called?
11:18:12 <zid> bcos_: 1
11:18:18 <zid> init adopts orphan processes
11:18:33 <Shockk> huh interesting
11:18:45 <klange> at no point should a process, besides init itself, not have a parent with a valid pid
11:18:47 <bcos_> That's stupid - init should terminated after the OS is initialised!
11:18:59 <zid> otherwise yea you'd get weird bullshit like thinking your parent was the wrong process
11:19:19 <zid> because pids are finite and get reused
11:22:06 <zid> bcos_: kill -9 1 should kill every process imo
11:22:33 <bcos_> That'd save a little memory..
11:22:52 <zid> The kernel is the important part of a PC, none of the other software can really do anything but waste cycles
01:19:47 <SopaXorzTaker> In a floppy boot sector, where does SP point to?
01:19:55 <SopaXorzTaker> like, before I initialize it manually
01:20:19 <SopaXorzTaker> I suppose it's at the end of the loaded bootsector, amirite?
01:20:31 <Mutabah> undefined I think
01:20:41 <klange> the end would be a very silly place to put it, but I believe the answer is "*shrug*"
01:25:55 <VegetableLasagna> SopaXorzTaker, probably to some random memory so interrupts won't fail. However, the first thing you should do is to set up your stack.
01:26:25 <VegetableLasagna> Some random safe place, I mean.
01:27:10 <VegetableLasagna> You don't need to turn off interrupts because for all CPU after the 8086, MOV SS, ... and the instruction following it are executed atomically.
01:27:38 <VegetableLasagna> Saved you two bytes there. :)
01:51:43 <oo_miguel> is the filedescriptor returned by syscalls in linux by open uniqe or it is sufficient if it is unique within the scope of a process?
01:52:15 <oo_miguel> I talk about the int value retunerd by the open syscall
01:52:24 <lachlan_s> They're unique within a specific process
01:53:15 <oo_miguel> ok. so after a fork when I duplicate the file descriptors the values seen by each process can stay, and I just adapt my internal kernel structures.
01:53:34 <lachlan_s> Yes
01:53:42 <oo_miguel> good to know, thank yuu
01:53:44 <oo_miguel> you
01:54:16 <lachlan_s> np
01:55:15 <VegetableLasagna> Mind you that file descriptors are not thread-safe in POSIX.
01:55:30 <VegetableLasagna> I mean you can make them thread-safe in your OS, of course.
01:55:51 <VegetableLasagna> But there's some undefined behavior in the specification which will be resolved in the next version of the specification.
01:55:57 <bcos_> They're not even process safe (no guaranteed file locks)
01:57:33 <oo_miguel> I understood that I can not even share one file descriptor between two processes
01:57:37 <oo_miguel> since they are local to a process
01:57:59 <VegetableLasagna> Right but he's talking about two processes opening the same file for writing simultenously.
01:58:02 <oo_miguel> and get dupliated on a fork
01:58:11 <VegetableLasagna> They may have different descriptors, that doesn't matter.
01:58:22 <oo_miguel> ok I understand that
01:58:27 <oo_miguel> they point to the same file
01:58:49 <oo_miguel> and there is no synchronization between the actions I perform on the file with the two open descriptros
01:59:41 <oo_miguel> but does this mean that reading and wirting to a file at the same time might get me some incomplete data ?
02:00:07 <VegetableLasagna> Of course.
02:00:17 <oo_miguel> a pipe for instance?
02:01:02 <oo_miguel> I would believe a pipe cares about guaranteeing that reads and writes are somehow atomic
02:01:24 <oo_miguel> but this will not apply to any file right?
02:01:51 <oo_miguel> s/any/every
02:03:11 <VegetableLasagna> Well, consider what happens when you attempt to read data from stdin.
02:03:48 <VegetableLasagna> Your process will just block waiting on that data until it has reached EOF.
02:04:38 <VegetableLasagna> That's not what happens with files on the disk because you *do* reach EOF --- no one knows someone is going to add more stuff to that file so blocking makes no sense unless you specifically bake it in.
02:04:56 <klange> VegetableLasagna: lol what's with the name?
02:05:10 <VegetableLasagna> It's a reference to Seinfeld.
02:05:11 <oo_miguel> VegetableLasagna: ok right I understand this EOF part
02:05:37 <oo_miguel> the read will just reach EOF and consider no more data will ever be added
02:06:31 <oo_miguel> but is THIS the "thread-unseafety" of the two file descriptors pointing to the same file?
02:08:02 <oo_miguel> becasue I am trying to understand what was said, that filedesciptors to one file are not "thread-safe" in POSIX
02:09:45 <SopaXorzTaker> How much stack does "int 10h, 0eh" eat?
02:09:56 <SopaXorzTaker> I allocated 32, then 64 bytes initially
02:09:58 <SopaXorzTaker> it ate them all
02:11:08 <VegetableLasagna> SopaXorzTaker, Why did you just allocate a couple of bytes? You have the whole memory at your disposal, with a couple of exceptions where stuff is reserved.
02:11:29 <SopaXorzTaker> well, indeed
02:11:31 <VegetableLasagna> oo_miguel, the problem I was alluding to is that if close() is interrupted by a signal, the state of the descriptor remains unspecified.
02:11:33 * SopaXorzTaker checks the memory map
02:11:35 * SopaXorzTaker is happy
02:11:53 <oo_miguel> VegetableLasagna: oh
02:12:10 <VegetableLasagna> oo_miguel, so if you want to close it again, you may either get EBADF (if it was closed) or you may end up closing a descriptor that was just open by another thread.
02:12:11 <oo_miguel> VegetableLasagna: ok thank you. I will keep that in mind
02:12:23 <oo_miguel> hmm
02:12:36 <VegetableLasagna> It's only safe to reissue close() if there are no other threads opening files.
02:12:47 <VegetableLasagna> One solution is to use locks but that's nasty.
02:13:07 <VegetableLasagna> And partly defeats the point of using threads.
02:13:16 <bcos_> Just postpone the signal until after the close has closed
02:13:45 <oo_miguel> I do not even have signals in my toy kernel yet
02:13:47 <VegetableLasagna> Well, an OS can do that. But POSIX doesn't guarantee it and different OSes seem to behave differently because of this.
02:14:03 <oo_miguel> but I also did not know about this "issue" in my real os
02:14:06 <VegetableLasagna> Linux closes the descriptor, as you said.
02:14:06 <oo_miguel> thanks for pointing this out
02:14:46 <VegetableLasagna> HP-UX fails with errno set to EINTR and then you have to reissue close().
02:15:00 <VegetableLasagna> The solution is that POSIX will introduce posix_close() as a safe replacement.
02:15:02 <oo_miguel> ok I undersntad
02:15:33 <VegetableLasagna> klange, do you /whois everyone or what? :)
02:15:41 <klange> only in channels i have ops in
02:16:31 <VegetableLasagna> Perhaps it's time to move on from Love4Boobies. :)
02:16:32 <oo_miguel> so with the regular close the file-descriptor might be in an corrupted state for some time, during the execution of the close?
02:16:38 <oo_miguel> is this how I can imagine it?
02:16:59 <oo_miguel> If i get context switched to another thread and examine it
02:17:09 <VegetableLasagna> It's not in a corrupted state. You just don't know whether it's open or closed.
02:17:24 <oo_miguel> do I know that I do not know it :)
02:17:26 <oo_miguel> ?
02:17:39 <VegetableLasagna> So you don't know whether to attempt to close it again because some other thread may have opened a file and got the same descriptor value.
02:18:04 <VegetableLasagna> Otherwise, closing a descriptor that doesn't exist is safe.
02:18:27 <oo_miguel> ok thank you
02:18:30 <VegetableLasagna> I'm not talking about the OS not knowing, I'm talking about programs.
02:19:00 <VegetableLasagna> The safe thing to do is to make your OS just close the descriptor and never fail with EINTR.
02:19:57 <lkurusa> good morning
02:20:24 <VegetableLasagna> Where POSIX goes wrong is that it makes it very difficult to write portable programs that function correctly (e.g., see Linux vs HP-UX). You have to use locks or avoid this risky thread stuff altogether.
03:38:01 <_mjg> anyone good with asm macros?
03:38:26 <_mjg> i want to concat a func name, but part of the name is *not* a value of any existing macro
03:49:15 <rakesh4545> can anyone explain how to set my segment registers for a bootloader which has to set GDT and load 32 bit C kernel?
04:37:51 <rakesh4545> hello
04:38:33 <rakesh4545> knock knock!
04:39:05 <tyler569> who's there?
04:39:35 <rakesh4545> wizard!
04:41:06 <rakesh4545> ok anta claus,
04:43:38 <rakesh4545> I wonder how to set my segments for the bootloader which switch to protected mode and loads a C kernel.
04:44:25 <rakesh4545> Also the string in C kernel appears to tangle up with other codes.
04:44:55 <rakesh4545> and on displaying the string I get weird characters on the screen.
05:05:01 <rakesh4545> rain1 how is your mikrokernel doing?
05:18:56 <rain1> im thinking of giving up.....
05:29:00 <rakesh4545> don't give up. I am writing a bootloader for your mikrokernel.
06:31:22 <glauxosdever> rain1: How is it going?
06:31:28 <rain1> fine
06:31:35 <rain1> how about you?
06:32:16 <glauxosdever> Gotta start writing the compiler (for my project). But I would still like to get involved in yours
06:33:25 <rain1> good luck with the compiler! that sounds cool
06:33:35 <glauxosdever> Thanks! :-)
06:33:45 <rain1> will it produce assembly/binary? will it be ELF or PE?
06:33:51 <rakesh4545> yes how is your compiler project going?
06:35:14 <glauxosdever> I haven't written a line yet (been somewhat busy with other things). But, assuming I have around two hours before 23:30, when I'll probably want to relax a bit before going to sleep, I could start with a lexer tonight
06:35:44 <glauxosdever> It will be ELF (initially), then I'll do PE and whatever format I invent/adopt
06:36:12 <rain1> i have written a compiler but it doesn't make assembly code just a bytecode
06:37:19 <glauxosdever> I'm thinking of producing machine code directly instead of assembly which has to be converted anyway to machine code.
06:37:30 <glauxosdever> It should be more efficient, I think
06:39:53 <glauxosdever> rain1: When will your microkernel project become parallelisable, as in two people will be able to work on two different things?
06:40:19 <rain1> that's a good question...
06:40:29 <rain1> i got a bit set back because i'm not sure whether to use GRUB or UEFI
06:40:42 <glauxosdever> It's probably something like a tree. When starting out, you have one single path, then there are features that can be developed one independently of the another
06:40:44 <rain1> both are sort of awkward
06:40:52 <rain1> yes true!
06:41:13 <rakesh4545> I am trying to write a bootloader program right now. But am stucked at one part.
06:42:39 <glauxosdever> rain1: THere is also using the BIOS and writing your own bootloader for it, but that's probably even more awkward. I think using UEFI would be best (if you can get past the imposing of the FAT filesystem and the PE format)
06:42:53 <rain1> ah
06:42:53 <glauxosdever> rakesh4545: Where?
06:42:56 <rain1> alright
06:43:29 <glauxosdever> But using GRUB is also more documented (that's why I used to recommend it initially)
06:43:57 <rain1> yeah its just i found it hard to go from 32 bit protected to long 64 bit mode
06:44:06 <rain1> so i kind of like how UEFI puts you straight into 64 bit
06:44:22 <rain1> but UEFI has some disadvantages too
06:45:49 <glauxosdever> I see two main disadvantages: being overall too complex and being quite Microsoft-centric (FAT and PE)
06:46:02 <glauxosdever> The "complex" part is however no worse than in BIOS
06:46:36 <rain1> yeah
06:46:42 <glauxosdever> And the "Microsoft-centic" part is only slightly worse than in BIOS (BIOS also was quite DOS-centric)
06:46:44 <rain1> and I have to pull in this GNU-EFI dependency
06:46:47 <rain1> or EDK II
06:46:55 <glauxosdever> But it imposed no FAT or PE
06:46:57 <rain1> basically building on top of this large framework of existing code
06:47:11 <johnjay> rain1: what gnu dependency?
06:47:21 <glauxosdever> It's probably mostly headers you can rewrite yourself
06:47:23 <rain1> it is a library
06:47:39 <glauxosdever> UEFI shouldn't require a library..
06:47:55 <rain1> oh
06:48:23 <yasar> There is special region in memory dedicated to VGA. Are there other such special regions. If so, where are they listed?
06:48:25 <glauxosdever> I have tried at some point to do something bootable with UEFI. I just wrote some headers myself
06:49:08 <glauxosdever> yasar: Why should that matter? Operating systems are supposed to be flexible (at least for architectures that are flexible, like x86 nowadays)
06:50:35 <yasar> I don't want to overwrite them accidentally
06:50:47 <glauxosdever> That's why memory maps exist :-)
06:51:08 <clever> yasar: i think it was called e820
06:51:41 <clever> ah, thats one of many methods: https://wiki.osdev.org/Detecting_Memory_(x86)#Getting_an_E820_Memory_Map
06:51:42 <bslsk05> wiki.osdev.org: Detecting Memory (x86) - OSDev Wiki
06:52:26 <glauxosdever> rain1: You probably should start from the EFI System Table. You have downloaded the UEFI spec, right?
06:52:38 <yasar> clever, thanks. That is exactly what I need
06:53:08 <glauxosdever> Or, if you don't mind, I can write some initial headers myself
06:53:30 <rain1> I was just reading it online
06:53:49 <glauxosdever> Ok, that also works
06:54:08 <rain1> glauxosdever: if using EFI then I will need to some extra bits to boot and test it in qemu
06:54:45 <rakesh4545> I don't know why but my C data are tangling with other code.
06:55:05 <glauxosdever> There is some UEFI firmware for QEMU, I forgot what it's called, is it Tianocore or something?
06:55:07 <rain1> i can make the PE toolchain
06:55:15 <rain1> yeah glauxosdever thats the one I was looking at
06:55:59 <glauxosdever> https://wiki.osdev.org/UEFI#Emulation_with_QEMU_and_OVMF
06:56:01 <bslsk05> wiki.osdev.org: UEFI - OSDev Wiki
06:56:06 <rakesh4545> qemu supports uefi?
06:56:25 <clever> rakesh4545: yes
06:58:05 <glauxosdever> rain1: So what's the plan?
06:58:25 <rain1> I think that doing all this UEFI stuff is the current plan
06:58:46 <glauxosdever> Alright. I can forget the compiler for tonight then :-)
06:58:48 <rain1> making a build script for the toolchain in osdev-toolchains, and making a barebones example
06:58:53 <rakesh4545> what are you building rain1
06:59:05 <asymptotically> rakesh4545: is something wrong with your linker script?
07:00:57 <glauxosdever> rain1: Interestingly, Debian already has a cross-toolchain for x86_64-w64-mingw32
07:01:11 <glauxosdever> Ubuntu probably too
07:01:18 <rain1> could that be used for building the UEFI bootable PE?
07:01:48 <glauxosdever> I think, yes. It's described in https://wiki.osdev.org/UEFI_Bare_Bones
07:01:49 <bslsk05> wiki.osdev.org: UEFI Bare Bones - OSDev Wiki
07:03:53 <glauxosdever> For the barebones example, you need the headers (either we write them from scratch, or we use gnu-efi)
07:08:59 <glauxosdever> rain1: So, may I start writing the headers for UEFI?
07:09:24 <rain1> yeah that would be good
07:11:13 <rain1> I'm trying to build the x86_64-efi-pe toolchain
07:11:34 <rain1> and I'll put notes about the debian and ubuntu ones, so people on those systems dont need to build the toolchain
07:11:55 <glauxosdever> What distro are you using?
07:13:21 <johnjay> glauxosdever: a cross toolchain for mingw32? is that just mingw?
07:13:44 <glauxosdever> Yes. It's what the osdev wiki suggests
07:14:04 <johnjay> ok. i thought you meant a toolchain to build mingw itself for a separate platform
07:14:40 <rakesh4545> asymptotically here is the complete repo https://github.com/rakesh491/boot-loader along with my build script.
07:14:40 <bslsk05> rakesh491/boot-loader - None (0 forks/0 watchers)
07:16:05 <rain1> ok I made the toolchain
07:16:09 <rain1> I use arch linux
07:16:35 <glauxosdever> Ok. I thought maybe by chance you used an Ubuntu derivative
07:17:24 <johnjay> rain1: how have you found arch for kernel writing so far?
07:17:41 <rain1> we are doing raspberry pi ARM & x64 UEFI
07:18:00 <rakesh4545> I am using Manjaro.
07:18:03 <rain1> the idea is that having 2 arches should help us not accidentally lock ourselves into a single arch
07:18:28 <rain1> bbl!
07:18:30 <johnjay> manjaro is based on arch though isn't it?
07:18:40 <rakesh4545> yes it is.
07:19:40 <johnjay> ah ok. i have debian and ubuntu and wanted to try compiling rain's kernel for my raspi 3b
07:19:53 <rain1> well its not mine, its a group project
07:19:58 <rain1> anyway, ill be back soon
07:23:35 <johnjay> oh ok
07:23:47 <johnjay> i thought rain was the owner of microkernel2018 repo
07:23:49 <johnjay> which has no files now lol
07:24:43 <bauen1> rain1: i'll be joining you after i've setup my server since that is currently draining all my available time
07:44:07 <glauxosdever> rain1: It turns out there are quite many definitions to do to be at least half-complete. I won't have that complete tonight :-(
07:46:08 <glauxosdever> We can still use gnu-efi for now, and then do some compatible headers to replace it. After all, as far as I can see, gnu-efi uses the same identifiers as the UEFI spec
07:56:43 <graphitemaster> Mutabah, how does Rust implement Option<T> in such a way that it's size is no larger than the T it uses
07:57:35 <graphitemaster> I was thinking maybe it uses like a union and wraps the initialization state with the T in memory but couldn't there be bit patterns of T that would conflict with that?
07:58:47 <graphitemaster> C++'s optional is always larger it appears than T
08:00:13 <graphitemaster> oh it's some stupid optimization when you combine with std::ptr::NonNull
08:00:30 <graphitemaster> it uses the null case for None internally on the pointer wrapped in the Option<T>
08:00:37 <graphitemaster> okay that confused me
08:00:41 <graphitemaster> not gonna lie
08:00:45 <graphitemaster> also very clever optimization
08:21:36 <lachlan_s> https://news.ycombinator.com/item?id=17945293
08:21:37 <bslsk05> news.ycombinator.com: I think, eventually, the whole stack will get rewritten from the ground up. Some... | Hacker News
08:21:42 <lachlan_s> Isn't that kind of beautiful?
08:26:24 <rain1> naiv
08:26:25 <rain1> back*
08:30:09 <rain1> bauen1: great! there is no rush
08:38:55 <rain1> I got tianocore OVMF working, can boot EFI stuff in qemu
08:42:15 <glauxosdever> rain1: Cool :-)
08:48:35 <rain1> yeah this is good progress! thanks for encouraging me to do it
08:51:12 <rain1> hmm
08:51:16 <rain1> *** Configuration x86_64-efi-pe not supported
08:51:22 <rain1> i cant build a gcc like that
08:51:29 <rain1> i wonder what the correct triple for gcc would be
08:51:34 <rain1> binutils build with that target
08:52:17 <rain1> maybe I can use x86_64-elf-gcc then x86_64-efi-pe-ld
09:11:06 <glauxosdever> https://wiki.osdev.org/UEFI_Bare_Bones#Prerequisites tells us everything we need to get started
09:11:08 <bslsk05> wiki.osdev.org: UEFI Bare Bones - OSDev Wiki
09:11:22 <glauxosdever> > You will need a GCC Cross-Compiler targeting the x86_64-w64-mingw32 target
09:14:21 <rain1> hmm
09:14:52 <rain1> im currently trying a slightly different thing than w64-mingw32
09:16:39 <glauxosdever> What target?
09:17:24 <rain1> x86_64-elf binutils and gcc, then building x86_64-efi-pe binutils
09:19:57 <glauxosdever> I think it's better to use a compiler that targets PE, instead of objcopying ELF to PE
09:24:57 <rain1> I cant build with mingw it seems, I guess that comes in a separate release than gcc/binutils from GNU
09:26:14 <glauxosdever> Anyway, gotta go
09:26:14 <glauxosdever> Goodnight!
09:26:38 <glauxosdever> I don't think it's meant to be built with mingw. It's just targeting it (where the same target can be used for UEFI)
09:26:51 <glauxosdever> Anyway, I hope someone steps in
09:26:57 <glauxosdever> gtg
09:27:38 <johnjay> lachlan_s: that seems incredibl idealistic to expect all operating systems to be obsolete and saas to take over
09:29:55 <lachlan_s> Oh, I agree, it totally is.
09:30:20 <yasar> I won't need a GDT if I don't intend to switch to protected mode right?
09:30:39 <rain1> https://gist.github.com/edwardw/7587876 I find it odd that they use pc-linux-gnu here
09:30:40 <bslsk05> gist.github.com: Boot to Rust in OS X · GitHub
09:53:28 <rain1> I don't understand why i'm supposed to use mingw
10:16:37 <johnjay> rain1: mingw-w64-x86_64
10:16:44 <johnjay> i have to type that an awful lot. :(
10:22:04 <oo_miguel> who and when should close the filedescriptors obtained via the pipe system call
10:22:19 <oo_miguel> two loose ends i close before calling execve and duping them
10:22:20 <mischief> it depends.
10:22:29 <oo_miguel> but when will usually the other two be closed?
10:23:16 <oo_miguel> after execve they are treated as stdin or stdout repectively. do c libraries close stdin and stdout at programm exit usually?
11:39:44 <jp> oo_miguel: usually you can think of file descriptors as just windows to some file description
11:40:02 <jp> closing a window doesn't usually affect what is under it unless it's the last reference or something
11:44:10 <Shockk> I have a question about `signal`
11:44:41 <zid> grats
11:44:43 * zid goes back to bed
11:44:45 <Shockk> lol
11:45:17 <Shockk> from what I understand, SIG_DFL SIG_ERR SIG_IGN should all be macros that expand to constant expressions with the function pointer type that signal takes as its second argument
11:45:48 <Shockk> but neither C nor POSIX seem to specify whether or not these should actually have a reasonable body, or if they're just symbolic
11:46:21 <zid> they should have type sighandler_t
11:46:21 <burzos> What does glibc or musl do?
11:46:36 <Shockk> good question I can check
11:46:56 <zid> https://www.gnu.org/software/libc/manual/html_node/Basic-Signal-Handling.html
11:46:57 <bslsk05> www.gnu.org: Basic Signal Handling (The GNU C Library)
11:46:59 <Shockk> zid: right, I'm wondering if they should actually do anything though, or if the purpose of them should just be for `signal` to know that they were passed in
11:47:08 <Mutabah> symbolic I think? e.g. one might be 0, another -1
11:47:19 <zid> Sounds more like it'd be up to you
11:47:26 <Shockk> hmm oh right
11:47:44 <Shockk> okay
11:49:36 <Shockk> I guess I'm still getting used to the idea that, if it's not specified, I can do it however I want
11:49:56 <Shockk> I do think symbolic makes sense thouhg
11:49:58 <Shockk> though*