channel logs for 2004 - 2010 are archived at http://tunes.org/~nef/logs/old/ ·· can't be searched

#osdev2 = #osdev @ Libera from 23may2021 to present

#osdev @ OPN/FreeNode from 3apr2001 to 23may2021

all other channels are on OPN/FreeNode from 2004 to present

http://bespin.org/~qz/search/?view=1&c=osdev&y=19&m=5&d=5

Sunday, 5 May 2019

12:07:43 <nepugia> I assume that it would kill me in any case :)
12:44:29 <w1d3m0d3> ARM will take over? sounds unlikely
12:53:20 <nepugia> but possible, i have seen one desktop board with an arm chip
12:53:33 <nepugia> though i also saw that one risc-v desktop board, so eh :)
01:08:33 <aalm> .theo
01:08:33 <glenda> That's the kind of comment that leads me to take bug reports less seriously in the future...
03:32:21 <hgoel> .theo
03:32:22 <glenda> And then we move on?
03:36:22 <hgoel> .theo
03:36:22 <glenda> words words words words
03:36:27 <hgoel> oof
04:11:37 <immibis> .theo
04:11:37 <glenda> SERIOUSLY??
04:28:28 <geist> blerb?
05:02:05 <doug16k> yay! I got the stack trace tracing all the way back to the user mode entry point (even when deep inside the kernel in a syscall) and it isn't messed up for one instruction, even during the weird periods where the stack is theirs temporarily
05:02:22 <doug16k> I love CFI directives!
05:07:16 <doug16k> https://github.com/doug65536/dgos/blob/master/kernel/arch/x86_64/cpu/syscall.S#L172
05:21:50 <doug16k> http://pasteall.org/pic/show.php?id=4156711144bfb50b92b930abb00f1b2b
05:39:07 <zid> doug16k quick, where is my mcfg table
05:44:11 <zid> oh right, I need q35 option duh
06:55:24 <ZetItUp> https://scontent-arn2-1.xx.fbcdn.net/v/t1.0-9/58864915_2790311354323191_5596902678056665088_n.png?_nc_cat=1&_nc_ht=scontent-arn2-1.xx&oh=b98e08d21f63256778bbd84d2dc871a7&oe=5D304706
06:55:25 <ZetItUp> heh
08:43:49 <geist> ZetItUp: what is it?
08:47:21 <bauen1> it's a harry potter meme
08:50:16 <klange> That's an old one - adding "Harry Potter and..." before research paper titles.
08:52:56 <geist> got it, just didn't want to generally click on things
08:52:58 <geist> so was asking
09:05:18 <immibis> geist: "Harry Potter and the Influence of Climatic Legacies on the Distribution of Dryland Biocrust Communities"
09:56:34 <jussihi_> If I push ds register to stack, how can I know it gets pushed there as a long?
09:56:44 <jussihi_> In gas I can just "pushl" but how about intel syntax?
09:57:20 <jussihi_> "push dword ds" does not work
09:57:40 <doug16k> jussihi_, it defaults to dword in 32 bit code
09:57:46 <jussihi_> ah great, thanks :)
10:05:22 <doug16k> if you wanted the opposite, then this probably does it: data16 push %ds
10:05:30 <doug16k> I never use intel mode though so unsure
10:05:58 <elderK> I love AT&T :D
10:06:10 <elderK> I really wish I could read a history of how the AT&T syntax come to be, why they chose the syntax they did.
10:06:13 <elderK> It'd be interesting as hell
10:07:33 <doug16k> ya I prefer AT&T too
10:09:48 <jussihi_> I prefer intel but I'm pretty bad in asm in general so...
10:09:59 <jussihi_> intel just seems more simplified
10:24:23 <elderK> jussihi_: I see it as the opposite: AT&T is very explicit, which makes it simpler for me to process. But YMMV :)
10:25:06 <elderK> Mostly I learned AT&T because I can use it for inline assembly and for standalone assembly. Of course, I didn't know you could use Intel syntax with GAS back when I learned AT&T syntax :D
10:25:23 <elderK> Back then, I was just glad that I didn't have to install something separate (nasm, yasm, whateve.r)
10:25:53 <jussihi_> Mmm I can imagine
10:26:10 <jussihi_> And yeah I agree that the explicity of it might make it "simpler" in cases like this :)
10:27:36 <elderK> jussihi_: There are a lot of nice features in other assemblers, though, features I wish were in GAS. Like, some assemblers for x86 allow you to define record structures and specify say, fields of those structures much as you'd expect.
10:28:00 <elderK> Where as in GAS, you end up needing to define those offsets yourself using macros, labels, .equ, whatever.
10:28:22 <elderK> Then again, I'm no AS guru so there may be ways to do this that I just don't know about.
10:32:14 <jussihi_> Hmm sounds familiar from making the multiboot bootloader :)
10:32:38 <jussihi_> I'm now trying to make the interrupt handling and trying to map all these interrupts to the IDT somehow
10:32:57 <jussihi_> Is there a way to index through the functions my ASM macro does?
10:33:16 <jussihi_> https://hastebin.com/iwomeyaruw.asm
10:33:51 <jussihi_> It now creates symbols for functions intr0-255
10:34:14 <jussihi_> but the function length is not static so I cannot just index them through for the IDT
10:34:46 <jussihi_> I don't wanna define all the "intr0(); , intr1(); ... " etc in the C code
10:35:38 <jussihi_> Can I somehow create a pointer list in assembly pointing to those asm functions so I could just walk through that list in C when adding those to the IDT?
10:37:53 <elderK> jussihi_: I'm afraid I don't understand that syntax.
10:38:19 <elderK> In general, though, you could forward each interrupt (handle exceptions a little differently), to another unified function.
10:38:27 <elderK> That function would record the registers, etc.
10:38:31 <elderK> Then it would call C.
10:38:44 <elderK> The C function could consult a table of function pointers - handlers, etc.
10:38:51 <elderK> Alternatively, you could transform interrupts into messages.
10:38:54 <elderK> And handle them that way.
10:39:24 <jussihi_> I was thinking that the ASM function would have its own number, corresponding to the interrupt number
10:40:02 <elderK> I've seen a lot of people create a set of asm-level interrupt handlers. These push stuff, as needed, then call another function. This second function saves a bunch of extra details, then calls some dispatcher in C.
10:40:07 <jussihi_> and it pushes it to the stack so that the C function knows which interrupt happened
10:40:33 <jussihi_> Now it just creates 256 different functions in ASM
10:40:46 <elderK> So, lots of stuff looks reasonably similar to what you've done. You don't need the full 256 functions in ASM.
10:40:48 <jussihi_> And I would need to define them in C to use them
10:40:55 <elderK> IIRC.
10:41:45 <jussihi_> I would like to stay far from this kind of code:
10:41:46 <jussihi_> https://hastebin.com/ludijiluwu.cpp
10:42:04 <elderK> ISR vs IRQ?
10:42:09 <jussihi_> and later:
10:42:10 <jussihi_> https://hastebin.com/tozininewu.c
10:42:23 <jussihi_> No I just mean code that I could do easier in a loop
10:43:01 <jussihi_> If I could just get a table of those ASM function pointers somehow, I could just loop through them when putting them to the IDT
10:43:08 <elderK> Yup.
10:43:22 <jussihi_> But is it possible in asm?
10:44:12 <elderK> I imagine so. I'd have to revise the SDM on interrupts and stuff to give you any better advice, I'm afraid.
10:44:28 <elderK> I'm sure you could do some neat tricks with linker scripts to generate a "IRQ table" if you needed it.
10:44:29 <jussihi_> Like i have these macros, so could I somehow add the function pointer to some separate list somehow when the macro is invoked during the assembly of the code
10:44:39 <jussihi_> Hmmm
10:45:14 <jussihi_> I'll try to disassemble it with IDA and see how it looks like
10:45:16 <elderK> You could for instance, loop through with your macro, defining these things. And switch to a different "segment" and store the address of what you added in the code segment or something, maybe.
10:45:35 <elderK> :) I haven't needed to do this in awhile, so I haven't invested much time and ingenuity into thinking of a nice way to do it.
10:45:43 <jussihi_> Yeah I was thinking about something like that
10:46:20 <jussihi_> holy moly, this 500-line asm code generates actually a 10kB binary
10:46:27 <elderK> Flat binary?
10:47:19 <jussihi_> Mmm
10:47:51 <jussihi_> Seems like I'm lucky
10:48:03 <jussihi_> All the resulting functions are 29 bytes in size
10:48:16 <elderK> I have a vague recollection of there being another way to retrieve the vector number, so that you didn't need to have a billion different functions
10:48:26 <jussihi_> So I could just loop from the first one
10:48:37 <elderK> That seems quite large for each.
10:49:03 <jussihi_> https://hastebin.com/wekonaduga.makefile
10:49:46 <elderK> Are these names generated?
10:49:54 <elderK> wekonaduga?
10:49:59 <jussihi_> yes
10:50:14 <elderK> I was going to say, that's some naming convention :P
10:50:38 <jussihi_> :D
10:50:56 <jussihi_> But I dont want to hardcode that 29-byte jump
10:51:07 <jussihi_> Hmm
11:12:55 <jussihi_> elderK: https://hastebin.com/adicifuyij.asm
11:13:43 <jussihi_> https://hastebin.com/hijevidode.asm
11:13:47 <jussihi_> here's the code
11:15:15 <jussihi_> Now I can just typedef it as a function pointer list in C and loop through it when adding those to IDT :-)
11:17:37 <elderK> Nice :)
12:12:50 <pterp> I'm trying to figuresomething out. When a microkernel boots up and loads the init program from the initrd, how does it communicate an error condition to the user? It can't have drivers for all platfornms, otherwise it's just a small monolithic kernel.
12:20:01 <bcos> pterp: For my design; boot loader uses firmware for console output initially, then cooperates with the next piece ("Boot Abstraction Layer") to figure out which video mode to use; then BAL starts a "display output module" and tells it to prepare for the selected video mode. After that, BAL tells boot loader to switch video modes, and (if that succeeds) the "display output module" takes over. Much later, kernel is started and the "display output module" gets
12:20:02 <bcos> converted into a process while it's still running.
12:21:20 <pterp> Ok. So the kernel will have to have a small display driver for each platform.
12:21:27 <bcos> ..then (much later again) the video driver is started and there's a hand-off ("display output module" process lets video driver take control of frame-buffer and starts sending its graphics to video driver)
12:23:11 <bcos> pterp: For my design; there's 2 main purposes: kernel doesn't have any video code at all, and (because display output module prepares before being expected to work) the chance of "undisplayable error" is almost zero (e.g. for things like "out of memory when trying to allocate back-buffer" the console is still being used)
12:23:51 <bcos> Note that kernel is mostly just a set of function calls. If other boot code calls a kernel function that returns an error, then the boot code displays the error and not the kernel.
12:31:45 <Aktive> hey
12:32:27 <Aktive> what do you guys think of fuchsia
12:32:47 <Mutabah> well, one of the devs hangs out here
12:32:55 <Mutabah> sounds like an interesting idea
12:33:35 <Aktive> if its open source it cant be dangerous to our privacy right?
12:33:53 <aalm> :|
12:33:57 <aalm> .theo
12:33:58 <glenda> whatever
12:38:40 <bcos> Aktive: Who said all of it will be open source?
12:41:30 <bcos> (I'd expect closed source firmware plus some closed source drivers, plus an unknown amount of closed source stuff in user-space; possibly with some kind of DRM scheme like the "secure enclave" thing)
12:41:35 <Aktive> nobody if its open source it cant be dangerous to our privacy right?
12:42:25 <Aktive> in any case thats not interesting
12:42:28 <Aktive> thats just scary
12:42:42 <bcos> If one of the pieces is open source, then that that piece can be dangerous to your privacy. If it is dangerous someone might fork the project if they can avoid copyright problems.
12:42:50 <aalm> fuchsia might need non-opensource blobs running in EL you can't reach/control
12:42:55 <Aktive> yes but at least we can look at it
12:43:29 <bcos> Hrm.. "if they can avoid copyright AND patent problems"
12:44:16 <Aktive> no i said If its open source
12:45:02 <aalm> why are you even worried about this, btw. ?
12:45:36 <Aktive> who
12:45:43 <bcos> "open source" is more than just the "anyone can do anything they like" licences (e.g. 2-clause BSD)
12:46:00 <Aktive> oh yes, if they use it on many kinds of devices
12:46:27 <Aktive> and there is no privacy its crap
12:47:26 <bcos> It's also possible to have something like secure boot, combined with an "anyone can do anything the like" copyright; where people can fork the project but the system will refuse to execute modified clones
12:49:12 <bcos> In fact; if you assume Fuschia will primarily target small/portable stuff (smartphones, tablets); then you can mostly expect that it'll be like Andriod now - e.g. OEMs will lock it down with their own stuff
12:49:36 <aalm> yep, some SoCs by samsung are kind of like that, they require a signed blob to boot the opensource os on top:/
12:51:08 <Aktive> no I thought that thats one of the reasons why they wanted to make fuschia in the first place
12:51:13 <Aktive> to have more control
12:53:19 <bcos> To have more control of what though? Google's business practices seem to be "create open source so other people adopt it; to promote the use of proprietary "google services"
12:53:22 <aalm> sure they have more control, and no privacy issues handing it over to the 'firmware', if necessary
12:54:08 <Aktive> updates, because the oem doesnt care much about that, only to sell
12:55:06 <Aktive> another topic why does microsoft not make their own desktop
12:55:13 <Aktive> *linux
12:55:49 <Aktive> like everything with the win 10 graphics shell, and being backward compatible, with windows and that you can use it like linux
12:56:09 <bcos> That's like asking why Microsoft doesn't want to go bankrupt
12:56:25 <Aktive> lol
12:56:33 <Aktive> but then they would choose to sell it
12:56:41 <Aktive> wouldnt people want that
12:57:08 <bcos> How many features do Linux distros provide to prevent piracy?
12:57:14 <aalm> you know apple went intel too?
12:57:19 <aalm> guess why
12:57:40 <Aktive> well they can keep the windows part closed
12:57:53 <aalm> and will, fear not
12:58:19 <Aktive> intel?
12:58:25 <Aktive> they use intel for years
12:58:38 <aalm> that's what i mean
12:59:09 <Aktive> yes the compatibility
12:59:39 <Aktive> but it went better for them didnt it
01:01:20 <Aktive> I be back later
01:11:56 <Aktive> but your right, business, business and business
01:16:39 <lukos> hello
01:17:39 <Aktive> hey
03:37:08 <epony> it's not for compatibility, they had no other choice, saying apple and compatibility in one sentence is ridiculous and naive
03:39:34 <epony> "compatibility" when they can't follow line ending, is absurd
04:07:20 <SomeT> whats a good diagram or documet/writings/article/video that explains an operating system kernel without reffering to an existing operating system?
04:09:12 <zid> intuition?
04:13:06 <SomeT> is this a company
04:13:10 <SomeT> or do you mean intuition?
04:13:32 <SomeT> I have all these books
04:13:33 <SomeT> https://wiki.osdev.org/Books
04:13:35 <SomeT> if it helps
04:14:37 <zid> so what's the actual problem you have
04:24:56 <SomeT> I want to understand what a kernel is and how it works
04:25:02 <SomeT> breaking it down kind of thing
04:25:17 <zid> the wikpedia article isn't enough?
04:25:22 <zid> which bit didn't you get?
04:27:21 <SomeT> which wikipedia article?
04:27:37 <SomeT> this one?
04:27:37 <SomeT> https://en.wikipedia.org/wiki/Kernel_(operating_system)
04:27:48 <SomeT> literally the diagram is to basic
04:27:48 <SomeT> https://en.wikipedia.org/wiki/Kernel_(operating_system)#/media/File:Kernel_Layout.svg
04:29:45 <bcos> SomeT: Can you draw a diagram showing steps involved in turning raw materials into a meal?
04:30:05 <SomeT> a meal has something to do with the kernel?
04:30:09 <bcos> No
04:30:10 <SomeT> or an actual meal relating to food?
04:30:15 <bcos> Food, yes
04:30:21 <SomeT> then no because I can't cook
04:30:45 <SomeT> but I think I get what you are indicating
04:30:55 <SomeT> so it is like, components/hardware feed into kernal
04:30:59 <SomeT> *kernel;
04:31:38 <bcos> Mostly, the diagram for one mean (roast beef & veg) is going to be very different to the diagram for a different meal (pizza); and the only diagram that makes sense for all the different cases is an extremely generic "input -> processing -> output"
04:31:44 <bcos> *for one meal
04:32:47 <bcos> For "kernel" it's the same - a detailed diagram for one kernel (e.g. L4) is going to look very different to a detailed diagram for a different kernel (e.g. Linux) and the only diagram that makes sense for most cases is extremely generic
04:59:11 <SomeT> so your saying I would just be better looking at an existing kernel to try and do this
04:59:16 <SomeT> makes sense, thanks
04:59:42 <bcos> I'm saying that if you want a detailed diagram you'll want to be more specific than just "kernel"
05:01:04 <zid> If you want to know how 'a kernel' works, various short descriptions like wikipedia are fine
05:01:13 <zid> If you want to know how a specific kernel works, look at that kernel
05:01:20 <zid> this applies to *any* topic
05:01:26 <SomeT> makes sense, thanks
05:01:40 <SomeT> I am just getting started on this, but usually visual representations make more sense to me
05:01:59 <zid> you're already done then, wikipedia provided a diagram
05:02:12 <zid> it sits between the hardware and the software and provides services
05:02:47 <SomeT> yes
05:03:10 <zid> http://www.makelinux.net/kernel_map/ The linux diagram is slightly more complicated
05:03:49 <zid> this is just an *example* setup of the linux kernel, also
05:03:55 <zid> 1% of all the actual code available
05:05:01 <SomeT> I am just aiming to get even a basic kernel coded at this point
05:05:13 <SomeT> but I wanted to break it down, somewhat, before starting that
05:05:39 <zid> software is iterative
05:05:43 <zid> you can't start at the end
05:05:55 <SomeT> true
05:12:16 <bcos> SomeT: Start by deciding what the OS will be used for (embedded, server, HPC, general purpose/desktop, ...); then decide the high level overview (e.g. monolithic kernel, micro-kernel or something else)
05:19:26 <SomeT> ok thanks
06:02:41 <stisl> I want to develop a testcase driven bootloader called "bootcheck" because it looks like that some things in my os are broken because of some bugs in the bootloader. For example I want to check if interrupts are triggered and so on some testcases which are needed in order to make sth. like a proof of concept that a custom OS can be booted.
06:03:03 <stisl> Does anybody now some important testcases?
06:03:30 <stisl> Some things that could be really important to check during boot?
06:04:59 <Vercas> Just run your OS in an emulator.
06:05:20 <stisl> Vercas, in the emulator everything is mostly fine
06:05:37 <stisl> but on the real hardware it looks not so good
06:05:38 <Vercas> Then you are not gonna have any more luck on anything else.
06:05:59 <Vercas> Most important thing you can do is making sure you have a solid DF handler.
06:06:04 <Vercas> If we're talking x86.
06:06:24 <stisl> what is a df handler?
06:06:29 <stisl> yes, x64
06:06:37 <Vercas> Double fault?
06:06:42 <stisl> ah, ok
06:08:45 <stisl> could it be a good Idea to handover everything with a table to the OS, so that it can be customized there if you want?
06:10:26 <heat> what do you mean
06:12:06 <stisl> I mean for example the GDT, interrupt handlers, a keyboard handler and so on things that could be good to handover to the OS
06:12:24 <heat> the OS is supposed to set all that up
06:13:03 <heat> the bootloader should only bootstrap itself and load your kernel onto memory
06:13:05 <stisl> yes,but if it isn't ready you have for eample an interrupt handler which can catch bugs which are there before the OS interrupt handler is set up
06:13:44 <heat> just keep your bootloader simple and you shouldn't need that stuff
06:14:41 <heat> bootloaders should follow KISS
06:14:50 <heat> (same with any hard-to-debug code)
06:16:45 <stisl> this should be, but a program without automated testcases is sometimes hard to test
06:17:10 <heat> using test cases in kernel development is very rare and not a good idea in general
06:17:45 <stisl> why?
06:18:29 <heat> because usually in kernel development you have lots of stuff that can break tests, or halt the whole machine, or even break/brick it
06:19:20 <stisl> that is true, but If you are testing carefully you can not break everything
06:19:36 <heat> yes you can
06:21:09 <heat> imagine you're testing your hard drive driver, somehow while testing you write a bad value to memory, the controller reacts in a bad/buggy way to it and boom, it's bricked
06:22:09 <stisl> I agree with that, but maybe some smoke tests could be really important
06:23:01 <heat> generally my smoke test is "does everything break?"
06:23:16 <heat> because most bad changes break user-space
06:24:55 <heat> whatever testing I do in the kernel is limited to stress-testing interfaces
06:28:12 <stisl> Sometimes the hardware is broken or has some bugs, for example the disk maybe you can catch it also with testcases in order to give some clear information
06:29:13 <zid> besides, the test-case for a kernel is often "multigigabit network switch while scheduling 47 processes in realtime while writing da.."
06:35:38 <stisl> In the embedded development some things can only run with privileged rights (becaues of the hardware e.g.) but it is common to have a lot of testcases
06:37:45 <stisl> mostly you have some connection to another machine in order to control it for example when the kernel crashes
06:38:44 <stisl> a serial interface could be good enough in order to make it simple
06:44:45 <heat> kernel testing != userspace testing
06:45:10 <heat> you can't(or shouldn't) use testcases because you simply won't have any use for them
06:45:47 <heat> most bad bugs come from edge cases that you won't even think of
06:46:30 <stisl> but how to catch them
06:47:17 <heat> debug
06:47:19 <heat> stress test
06:47:39 <heat> use what you're debugging
06:48:33 <stisl> this I am doing currently, it feels for me that it is not enough
06:51:33 <heat> most issues aren't obvious
06:51:39 <heat> it's just up to you to find them
06:51:43 <heat> patience is key
06:57:14 <stisl> In my experience the simulation should be good enough to test if sth. harder or conceptional is broken, but only with simulating it is hard to catch the other bugs
06:59:07 <stisl> sometimes the debug interface isn't available or good enough to check what is going on, in this case testcases can help
07:00:24 <stisl> In some companies I have worked for it was only allowed to run the testcases on the real hardware
07:00:54 <stisl> As a system tester I was there
07:03:33 <zid> I'd love to see a testcase for the swapgs bug
07:03:47 <zid> I think people who are "into" test cases come from non-compiled language backgrounds
07:04:03 <zid> the class of mistakes test-cases catch are the same class that type systems resolve, with a major overlap, imo
07:04:31 <stisl> sometimes people also have too much testcases or too big ones, this is what I have also seen, then it is hard to say what actually is broken
07:05:06 <zid> and that isn't even accounting for the fact that test cases are just as likely to be wrong as the code they're testing, in the case of a typed language
07:05:19 <zid> because they have to test behavior, not fickle language details
07:05:33 <stisl> sorry,I can't follow
07:05:44 <stisl> give an example please
07:06:00 <zid> swapgs bug
07:06:11 <zid> I can't even begin to *imagine* how you'd detect a bug like that programatically
07:06:31 <zid> and the code to test for it would be harder than the code that just does it correctly in the first place
07:06:38 <bcos> "cmp [gs:0],magic_value"
07:07:11 <geist> well, you can test that the instruction works, but i'm not sure what that's supposed to prove
07:07:26 <heat> inb4 spectre testcases
07:07:34 <zid> bcos: and how do you perform that test on the instruction after a double fault, specifically?
07:07:48 <zid> massive test rig? Now we're more complicated than just checking if the code is correct by hand
07:08:02 * bcos was thinking more like "cmp [gs:0],magic_value; jne oh_crap_kernel_did_swapgs_twice?"
07:08:16 <zid> yea, it makes *way* more sense as an assert
07:08:18 <zid> than as a testcase
07:08:23 <zid> And I feel that's true for 99% of a kernel
07:08:38 <heat> right
07:08:40 <zid> (and 80%+ for any typed language at all, pulling numbers out of my ass)
07:09:02 <heat> there's no memory protection in the kernel so lots of things can go wrong and it's not necessarily your fault
07:09:36 <bcos> Well; micro-kernel using Intel's "software guard extensions" might have a little memory protection.. ;-)
07:09:57 <zid> I'd also love to see your testcases for TXT etc :P
07:10:12 <zid> without just testing that the cpu isn't faulty
07:10:38 <bcos> Heh - the number of times Intel screwed up TXT I think the test case would be "Did I accidentally use it? {No/panic}"
07:10:46 <zid> hah
07:11:13 <heat> what's txt?
07:11:22 <bcos> Transactional extensions
07:11:56 <heat> oh right, that thing
07:12:03 <geist> as a side note i *highly* recommend assrting up the kernel like crazy. it as paid off dividends in my stuff
07:12:08 <stisl> how do you test it by hand?
07:12:14 <heat> i thought that was tsx
07:12:32 <heat> stisl: "Does it break? If yes, debug. If no, proceed carefully"
07:12:35 <geist> i assert left and right, provide a way to compile them out, etc. catches hugh swaths of problems prior to them going completely wild, makes debugging far simpler
07:12:40 <heat> where carefully = go wild
07:14:59 <geist> side note. i got to reading this last night, it's very nicely done htmlification of a the cray-1 manual: http://ed-thelen.org/comp-hist/CRAY-1-HardRefMan/CRAY-1-HRM.html
07:15:36 <zid> And my web browser probably has to use more processing power than a cray had, to render that page :p
07:16:38 <geist> oh by a long shot, that being said an 80Mhz 64bit vector machine in 1977 was still pretty advanced
07:16:43 <bauen1> i have asserts everywhere, but don't allow to compile them out, since most of them are "if this fails, we're beyond fucked up anyway"
07:17:01 <geist> bauen1: yah for that i tend to split it into always on asserts and debug asserts
07:17:02 <zid> That's why you should be allowed to compile them out
07:17:16 <zid> asserts are pretty much definitionally 'never triggerable'
07:17:24 <geist> debug asserts are most of them. they're things like checking magic numbers or validating that arguments to internal routines make sense, etc
07:17:53 <geist> what it generally helps me do is catch problems earlier in the development cycle. much easier to debug 'the thread structure is corrupt' than the system just triple faulting
07:18:01 <zid> yup
07:18:52 <geist> that being said, aside from code size increase, on modern machines judicious use of asserts doesn't seem to affect performance that much
07:19:10 <geist> especially since most of them are inside fairly high level routines that are already expensive
07:19:24 <geist> the branch predictor usually works through those well
07:19:24 <bauen1> well, and the compiler might decide their useless anyway
07:19:32 <geist> that doesn't happen, no.
07:19:38 <bauen1> yes
07:19:55 <geist> well, i haven't seen that sort of thing happen much
07:20:00 <zid> It can if your asserts aren't implemented correctly and the compiler can prove the code is correct
07:20:04 <geist> as in the type of things i'm asserting for the compiler can't elide
07:20:06 <bauen1> __attribute__((nonnull)) void f(void *a) { assert(a != NULL); /* <= gcc will optimise this away */ }
07:20:21 <bauen1> which i find kind of stupid but yes
07:20:25 <zid> how is that stupid
07:20:31 <zid> you specifically annotated it that it's impossible
07:20:44 <geist> okay sure, but that's also one of the reasons that attribute is a teensy bit dangerous sometimes
07:20:52 <bauen1> yes, but gcc will optimise *EVERY* error checking away
07:20:56 <geist> it's not bad, but i haven't found it to be as useful as i thought it'd be when i first learned of it
07:21:02 <heat> C is supposed to easily interface with other languages
07:21:04 <zid> impossible things it doesn't matter if it does
07:21:09 <bauen1> true
07:21:19 <zid> that's why it can, and do does it
07:21:21 <zid> (lovely sentence)
07:21:31 <geist> yah i think that's a bit of a straw man. yes you have a case where thec ompiler optimizes something away
07:21:42 <geist> but that does not in any way take away from all the places where it doesnt
07:22:07 <geist> super aggressive LTO I might see it trying to elide more stuff
07:22:19 <zid> yea I love super aggressive LTO, gives you much better diagnostics :P
07:22:21 <geist> which i think is fair if it comes out that way
07:22:31 <zid> especially wrt uninitialized uses of struct members
07:22:50 <zid> something idiomatic like struct bob b; bob_init(&b); is technically a pointer to an uninitialized wotsit
07:22:53 <geist> but most of the really good asserts are things like 'did you actually pass me a pointer to something that isn't fucked' or 'did the linked list i just traverse end with a dangling pointer' or whatnot
07:23:00 <zid> and without lto it may be impossible to check
07:23:21 <geist> sure, but in that case bob_init() can assert the opposite. assert that the structure *isn't* initialized
07:23:37 <geist> since it's explicitly taking a bag of bits and making it into someting
07:23:48 <bauen1> zid: the issue is really, that gcc will happily optimise away error checking of your library and if you then compile a programm against the library that doesn't know about nonnull it will happily call it with NULL (and the error checking includes if-elses), so nonnull is just dangerous
07:24:04 <zid> It doesn't happily optimize away error checking
07:24:09 <bauen1> yes
07:24:10 <zid> it optimizes away *impossible* errors
07:24:10 <bauen1> it does
07:24:14 <bauen1> oh true
07:24:17 <geist> i think you're taking it a bit far there bauen1
07:24:28 <geist> because it may optimize away one thing does not mean 'it optimizes away error checking'
07:25:12 <zid> If my compiler didn't optimize out if(sizeof(int) == 47) printf("This cpu is unsupported"); on x86 I'd be very very upset
07:25:31 <zid> that string should only appear in the binary *if* sizeof(int) is 47, and always print it, and the if() removed :P
07:25:41 <geist> yah if (false) { please dont actually do this but check that it compiles anyway; } is a very common pattern
07:26:07 <zid> everything else is useless inanity
07:26:17 <bauen1> alright
07:26:40 <bauen1> it does mean that __attribute__((nonnull)) is useless in it's current form for anything that takes user-input in anyway :/
07:26:40 <geist> but.... null pointer optimizations are a set of optimizations that the compiler does that *does* cause problems sometimes
07:26:50 <zid> it's.. useless if it can be null
07:26:51 <geist> there's a switch to turn it off, but it then throws out the baby with the bathwater
07:26:59 <zid> because that's a *promise* to the compiler you will *never* pass it a null pointer
07:26:59 <zid> ever
07:27:05 <zid> under any circumstance whatsoever
07:27:31 <zid> The exact same logic applies to restrict and other similar optimization paths
07:27:44 <geist> there's a semi related one i've bumped into: passing enums as arguments and then testing for out of bounds
07:28:03 <geist> i've seen i think clang elide the check, because it's an enum, and thus the only valid values are in the enum
07:28:06 <geist> which is dangerous
07:28:28 <zid> for example, strtol(a, &a, 10); is actually *UB* because you promised not to pass the same parameter twice
07:28:39 <zid> not just a funny bug in libc
07:28:40 <geist> especially since the caller can jam in any random int
07:29:00 <zid> Turns out C has semantics, and you have to abide by them if you want sensible output
07:29:11 <zid> If you're trying to bend the rules, don't be upset when the compiler bends the output :P
07:29:54 <geist> at some point when i first learned of attribute nonnull i started trying to apply it everywhere. i think it can work for internal code to a project, but i'm not sure i'd use it on external facing stuff
07:30:13 <geist> or maybe if you can only use it on a separate set of externally facing headers to try to get the caller to be good, but not modify your codegen maybe
07:30:44 <zid> Yea you'd pretty much only use it for internal functions after checking external functions for correctness
07:30:55 <olsner> iirc the nonnull attribute only works in plain C, that's the main reason I've never been able to use it
07:31:02 <zid> think of like, png_init(png *p){ if(!p) return; png_init_colourspace(p); }
07:31:05 <geist> i was essentially using it for 'this' pointers to OO looking C code
07:31:11 <zid> the inner function can be attribute nonnul if that's the only callsite
07:31:28 <bauen1> you can only use it for internal stuff, otherwise gcc will remove any checks against null, so you should wrap that
07:31:30 <zid> so that you can benefit from an LTO style optimization even without LTO
07:31:34 <heat> how is it even useful
07:31:53 <zid> can't get LTO optimizations between modules, usually, for example
07:31:56 <heat> as far as I can see it's useful in that it warns you against passing NULL as arg
07:32:08 <geist> here's where the language (C++ at least) slaps you and i've been burned by: operator new shall not return a nullptr according to the language
07:32:20 <geist> and so the compiler will elide all null pointer tests downstream of a new
07:32:27 <zid> you get an exception instead if oom I take it
07:32:33 <zid> s/if/of
07:32:39 <geist> right, *even* in the nothrow case the logic is the same
07:32:49 <zid> no idea what nothrow is, I stick to sane languages :P
07:33:27 <geist> so, in kernel code or whatnot where you may be using new you end up having to do some dumb tricks to avoid it. basically declaring your own special new with a nonstandard arg seems to defeat this
07:33:42 <heat> or
07:33:51 <heat> pass -fcheck-new to g++
07:34:26 <geist> hmm, question there is what does it codegen
07:34:33 <heat> works wonderfully and you can keep using new as usual
07:34:39 <geist> i dont want it to *check*, i want it to assume it could be null
07:34:57 <olsner> hm, with nothrow new there should be a nullcheck generated around the constructor call
07:35:02 <heat> what? isn't that the same?
07:35:49 <geist> well, check seems to imply it'll emit some code... what does it emit? a call to abort() if it's null?
07:36:09 <heat> probably doesn't touch the pointer if it's null
07:36:32 <zid> 'probably' is a good answer :P
07:36:41 <heat> godbolt is the answer
07:36:46 <zid> not really
07:36:53 <zid> because it may do different things under different cirumstances or cpus
07:37:25 <geist> hmm, unclear: https://gcc.godbolt.org/z/TvySQ9
07:37:28 <geist> doesn't seem to do anything
07:38:05 <geist> https://gcc.godbolt.org/z/rAKGDU
07:39:17 <heat> https://godbolt.org/z/4u1PX0
07:39:20 <heat> works here I think
07:39:56 <heat> yeah it works
07:40:13 <geist> huh. interesting
07:40:27 <geist> i guess the difference is in your case you tested based on it
07:40:40 <geist> but in my case since i went ahead and used it anyway it didn't bother
07:40:41 <heat> probably because you're constructing an int without a value
07:40:57 <geist> well, i made a later one that sets it to something
07:41:02 <heat> if you did new int(5); it might work
07:41:27 <heat> yeah
07:41:31 <heat> that's it
07:41:55 <geist> but doesn't test. so it implies it does what i originally wanted: it removes the assumption that it's nonnull
07:42:33 <geist> oh hrm. that's interesting: new int(5); does generate a test, but *a = new int; *a = 5; does not
07:42:44 <heat> (at least on x86) fcheck-new means "test implicitly if you're constructing and assigning values and don't optimize null tests"
07:42:51 <zid> s
07:42:57 <zid> wrong window, I meant to go south, elsewhere
07:43:23 <geist> it's really the 'dont optimize null tests' that i want
07:43:27 <geist> heat: thanks! I could use this
07:43:37 <heat> No problem :)
07:44:06 <geist> well... actually it's not useful. clang doesn't seem to honor it. sigh.
07:44:17 <geist> it takes the switch but it doesn't seem to mean exactly the same thing
07:45:18 <heat> :/ seriously
07:45:30 <geist> it's clear it still optimizes the null test: https://gcc.godbolt.org/z/6eNj5j
07:46:16 <heat> yeah
07:46:27 <geist> aaah but if you use the std::nothrow version...
07:50:22 <heat> geist: don't you fuchsia guys work closely with the llvm and clang projects?
07:52:39 <geist> indeed. which is why it being not supported in clang means either we'd have to add support, or it's not going to be useful
07:52:48 <geist> since clang/llvm is clearly the direction that we're going in
07:52:52 <geist> for better or worse
07:53:06 <zid> It's sadly the direction the universe seems to be going
07:53:33 <zid> All things leading to entropy and all that
07:53:34 <heat> i mean fcheck-new might be useful enough
07:53:48 <geist> yah as a compiler i'm not as onboard as lots of folks. it's pretty good, but i still slightly prefer gcc for the type of coding i do (low level, bare metal, etc)
07:54:02 <heat> geist: can you even compile fuchsia with gcc?
07:54:04 <zid> Yea gcc is a nicer tool, but llvm has some lovely features
07:54:12 <zid> and nicer ecosystem
07:54:18 <geist> heat: yeah we have and still do compile all of zircon with gcc and clang
07:54:30 <geist> the rest of the fuchsia never got compiled with gcc
07:54:34 <heat> oh thats nice
07:54:55 <geist> but the toolchain folks generally prefer that we drop gcc, since they have no itnerest in supporting it
07:55:15 <heat> what exactly is the difference?
07:55:28 <heat> aren't they just C/C++ compilers?
07:55:29 <geist> so eventually that will happen. early on we could only compile zircon with gcc because clang/llvm support for more esoteric things like complex linker scripts and whatnot weren't there
07:55:41 <geist> most of the problems we did have weren't in clang. they were in llvm
07:56:01 <geist> and specifically in the linker and whatnot. lld was fairly basic a few years back. or at least was written to largely compile apps
07:56:14 <geist> didn't seem to have anywhere as much flexibility in binary layout and whatnot as binutils
07:56:36 <geist> things like objcopy and objdump are still somewhat ahead of the llvm equivalents
07:56:58 <geist> but, for fuchsia we got lots and lots of bugs and features added to llvm to bring it up to parity with binutils
07:57:09 <geist> some of it was also backing off more esoteric stuff in the linker scripts and whatnot for kernel code
07:57:15 <clever> and last i checked, can llvm even do IO? i dont think the IR can represent IO actions, so you need to link against some raw assembly?
07:57:29 <geist> IO as in in/out instructions?
07:57:35 <clever> yeah
07:57:43 <geist> just use inline assembly for that. same as gcc
07:58:09 <clever> does that still persist thru the IR phase?
07:58:15 <geist> beats me.
07:58:33 <clever> https://llvm.org/docs/LangRef.html#module-level-inline-assembly
07:58:36 <geist> it works in that the code it generates seems to be there
07:59:04 <geist> for the most part my experience is clang's inline assembly support is just about as good as gcc. the built-in assembler is a teensy bit more picky about things, especially on ARM
07:59:11 <geist> but otherwise it works pretty fine
07:59:19 <clever> ah
08:00:19 <geist> clang codegen in my experience on x86 and arm64 is a little wonky. it's on the average pretty good, about on par with gcc for a large, mostly memory bound thing like a kernel
08:00:44 <geist> the more you zoom in on any given piece of codegen the more strange it gets. sometimes clang seriously misoptimizes things but on the average it does a good job
08:00:53 <zid> gcc still smashes it in some areas, if you're in that area, clang is no-go
08:01:05 <geist> if it happens to seriously misoptimize something performance critical then its a problem, but then most code in most 'regular' code doesn't matter that much on modern hardware
08:01:17 <zid> I would say in most software in general, to boot
08:01:46 <geist> yah. stuff like kernels tend to be largely load/store bound, lots of decision trees, less room for a lot of really meaty optimizations anyway
08:02:05 <heat> but I mean, why llvm vs gcc?
08:02:10 <zid> what other option is there
08:02:10 <heat> that's just what I don't get
08:02:23 <geist> in our case it's because we're google, and google has people on staff to work on clang/llvm
08:02:25 <heat> why would you ever need to switch toolchains to llvm?
08:02:38 <geist> and.. quite specifically there are all sorts of side things being developed in llvm
08:02:53 <geist> asan, safe stacks, safe call stacks, etc
08:03:11 <geist> lots of code gen changing safety things and whatnot that toolchain folks love to play with
08:03:30 <geist> sicne llvm is a big pile of modular code then the idea is to centralize all your efforts there
08:03:39 <geist> that's where llvm will win in the end, by simply being more flexible
08:04:03 <mahackamo> i thought ICC beats both gcc and llvm
08:04:26 <mahackamo> why does everyone pretend theres only two options
08:04:28 <geist> also things like clang code formatter, which uses the actual clang front end to format code, and stuff like clang-tidy which can do some fairly hard code code analysis
08:04:46 <geist> mahackamo: ICC is intel only, only works on intel cpus. why does everyone pretend there are only intel cpus?
08:04:48 <heat> icc is proprietary
08:05:05 <heat> you need to pay for icc too
08:05:11 <mahackamo> does llvm not allow proprietary plugins?
08:05:27 <zid> I'd say sort of ideally it'd compile witha ll compilers just for coverage
08:05:29 <heat> I honestly don't care about the proprietary thing, just the cost
08:05:40 <geist> probably could, but then intel would need to provide it. AMD has their own thing, could be they've switched to llvm
08:05:53 <geist> ARM just focuses their efforts on gcc and llvm, which IMO is the right idea
08:06:48 <geist> but that being said, as a result of competition i think gcc has upped their game. as it should. competition is good
08:08:24 <zid> gcc definitely seemed to have stagnated a bit between 5 and 7
08:09:19 * heat sees C/C++ compilers as really boring monolithic things
08:09:38 <zid> seemed*
08:10:38 <geist> that's hwo i've traditionally viewed it. i usually favor stability, reliability over bleeding edge
08:10:52 <geist> but, at least at google, i've hit exactly the opposite mentality of the toolchain team
08:10:59 <geist> at once it's refreshing and infuriating
08:11:24 <geist> as in they have no trouble running bleeding edge, new build every week, etc. the idea is if you always do that and always run aggressive test suites you'll always have the freshest stuff
08:11:37 <geist> and, well, it basically works. it goes against everythign i've ever known about compilers
08:12:03 <heat> I mean, I like bleeding edge but I really can't see how you can innovate in C/C++ compilers, which are really stable languages
08:12:08 <geist> this is why they dont want to touch gcc, they can't operate that way on that codebase
08:12:27 <geist> they're mostly innovating with extra features. ASAN, etc
08:12:48 <geist> plus remember clang/llvm are kind of a big codebase. you may have to rebuild the whole thing even ifyou're ust adding a feature for a side project
08:13:13 <geist> may be that the front end of clang doesn't really get any updates for a while but the back end llvm side is getting lots of tweaks continually
08:13:24 <geist> but you rebuild the whole thing
08:15:49 <zid> writing it in C would have fixed that :P
08:15:57 <geist> but, like all things, that's a very Big Company solution to things
08:16:15 <geist> as in you can spool up a whole team, run thousands of test suites, have mega workstations to recompile stuff
08:16:32 <geist> (clang/llvm takes a *long* time to compile even on a very high end workstation)
08:17:24 <geist> but then the cool thing is in as much as features and bugfixes happen as a result of a few big companies wanting to maintain it, everyone benefits
08:17:30 <geist> so can't really complain that much
08:47:59 <geist>
08:55:18 <geist> huh. that's cute. the task switching (and irq) mechansism in the cray-1 is like a simpler TSS
08:55:46 <geist> basically it's a 16 word in memory thing that is the full state of the cpu. you can only exchange the current registers with one of them
08:56:15 <geist> to trigger an exchang eyou can either do it manually (a few opcodes for 'program exit') in which case it returns to the task that exchanged to you in the first place
08:56:27 <geist> or via an interrupt, which automatically exchanges with a dedicated one
08:56:45 <geist> so iframe or anything, it only can switch contexts via one of these things
08:57:35 <geist> it doesn't save which one you're currently running, it remembers (in a hidden register) the task exchange memory you came from, that's how you'd syscall or exit back to the kernel
08:58:45 <geist> it saves all the integer registers and the BA and LA (base address and limit address) register, which are a physical address + size of the task
09:10:40 <heat> I've got a question: It turns out that I'm getting a lot of variance in my timer calibration code in qemu. Is it possible that it's getting delayed because of later vmexits?
09:11:35 <geist> yah
09:11:44 <heat> since I'm calibrating the PIT for 10ms one-shot first, then I'm writing UINT32_MAX to the lapic's initcnt and doing rdtsc
09:11:58 <geist> every time you touch a registers assume there's going to be at least a few thousand cycles gone
09:12:14 <geist> a hardware register that is. via io or mmio
09:12:21 <geist> it tends to skew low level timing pretty badly
09:12:38 <heat> the weird thing is that I get distant values when I reset qemu
09:13:00 <geist> note that you can generally directly read this information from a cpuid leaf that it will export
09:13:29 <geist> around cpuid 4000.0000 there's a few leaves that hypervisors tend to expose with information like how fast the ticker is running and whatnot
09:14:37 <heat> hmm
09:14:49 <heat> that's not exactly my problem though
09:15:08 <geist> yah i know. just saying calibration is a complicated enough problem while being virtualized
09:15:10 <heat> the thing is that the delay caused by the vmexits is changing wildly
09:15:21 * geist nods
09:15:25 <heat> hmmm
09:16:21 <heat> is there a solution to this problem?
09:18:33 <heat> I mean I could loop it and get an average
09:18:49 <bcos> Which timer are you calibrating?
09:19:01 <zid> by average I hope you mean like min or mode or something
09:19:07 <zid> else you're just smearing out the error you know is there
09:19:37 <zid> linux wouldn't boot on one of my machines for a long time, because you'd get keyboard irqs during pit calibration
09:19:40 <heat> bcos: lapic and tsc
09:20:33 <bcos> Could try increasing time PIT is configured for (would reduce impact of VMEXIT at the final RDTSC)
09:20:50 <doug16k> I wouldn't recommend expecting an IRQ for calibration
09:21:06 <doug16k> poll the timer for a short time (milliseconds)
09:21:26 <heat> ah
09:21:30 <doug16k> it doesn't have irq disabled when it is doing that?
09:21:31 <heat> maybe the IRQ is worsening it
09:22:00 <heat> doug16k: irqs enabled, everything masked except the pit irq
09:22:36 <zid> Thank god for HPETs
09:22:59 <eryjus_> heat, you don't have something else generating periodic irqs do you?
09:23:08 <heat> no
09:23:56 <heat> only irq0 is unmasked, which is reserved for the timer so I don't believe so
09:24:13 <heat> I also don't enable any other device's irq before calibrating the timer so yeah
09:24:39 <doug16k> I try to use PM timer first of all. if that fails then I program PIT channel 2 with a count and poll until the count runs out
09:24:53 <doug16k> 1ms is the duration I use
09:25:02 <heat> I mean I could do that
09:25:13 <eryjus_> i just went through this exercise recently and I did have a bug where my timer wasn't timing and threw off the calcs.. sounds like you have the opposite problem though
09:25:56 <heat> gtg i'll be back in like 20 min
09:25:57 <eryjus_> and i ended up polling for 10ms to calibrate
09:26:03 <heat> i'll think of solutions in the meanwhile
09:26:34 <doug16k> PIT "nsleep" as I described has about 838ns resolution
09:26:59 <doug16k> 1ms is all day
09:28:43 <doug16k> if you want extremely precise calibration, go 10ms
09:36:15 <doug16k> the I/O instructions to poll are likely to take about a microsecond, so I would expect it to be +/- 2us at best
09:37:11 <doug16k> leaning heavily toward + side
09:40:13 <doug16k> in my implementation I actually program the count with 64 extra counts and wait for it to go at or below 64. then I can look at the last count and return approximately how much time actually elapsed
09:40:51 <doug16k> i.e. I can see how far over it went, up to ~53.6us
09:40:52 <zid> yea, I was about to say on the -/+ thing, if you take the average of how long your polls are taking, you can bail early just as easily as late
09:41:41 <zid> if you know you take 32 units to poll and you hit 9999/1000 units you should probably stop eh
09:42:24 <doug16k> yes, ideally
09:42:39 <doug16k> depends on whether caller is giving you a lower bound
09:43:07 <zid> might be worth spinning, at that point, even then
09:43:13 <doug16k> nearest would be okay, but as a delay API it will make people add more just in case
09:43:39 <doug16k> my nsleep is defined as "sleep at least this long"
09:43:43 <doug16k> it's allowed to go a bit over
09:43:53 <zid> how accurate would rdtsc be here
09:44:00 <zid> for a spinloop to get the extra 'tick'
09:44:07 <doug16k> super accurate. but this code is what I am using to calibrate rdtsc :)
09:44:15 <zid> yea I mean for the nsleep case
09:44:19 <zid> where you'd rather not be early
09:44:41 <zid> sleep using the pit/hpet/whatever at 800ns accuracy, then spin for a bit on 10ns accuracy or such
09:44:52 <doug16k> my algorithm is: program timer with a large count, read rdtsc, use PM timer or PIT to delay 1ms, read timer, read rdtsc, look at how much time really elapsed, calculate timer and rdtsc freq
09:45:10 <zid> I wrote an OS level sleep loop like this for a game once
09:46:04 <zid> OS interface level, I guess, how would you phrase that?
09:47:16 <doug16k> delay API I guess
09:47:20 <doug16k> no sure what you mean
09:47:36 <zid> The collective term you'd give for the set of apis userspace provides to you
09:47:44 <doug16k> microsleep or nanosleep or usleep or nsleep are common names
09:47:56 <zid> The system apis, I guess
10:03:46 <doug16k> ah as in the libc interface
10:05:42 <zid> or syscalls, just not-internal-to-kernel
10:05:51 <zid> I'm happy enough with "system api"
10:07:29 <doug16k> linux has usleep, sleep, nanosleep in libc
10:07:39 <zid> yes, I just wanted a term
10:07:46 <zid> I know what functions are available
10:08:08 <zid> I just could not for the life of me think of the words to use for "the things you can call from userspace collectively, that are provided by others"
10:08:42 * mahackamo #define's _GNU_SOURCE
10:09:48 <mahackamo> you guys every had to wrestle with that labrynth of preprocessor XOPEN/BSD/GNU define mumbo jumbo
10:11:11 <heat> mahackamo, no
10:11:40 <mahackamo> ah yes c99 probably shields your eyes from the cold truth
10:11:41 <heat> If I ever need one of those defines I just define everything and that's it
10:11:55 <heat> ah, you wish
10:11:57 <heat> c11
10:12:05 <heat> the superior C
10:12:36 <zid> C89 with anonymous structs, the superior C
10:13:22 <heat> C++17, the superior C
10:13:24 <zid> I'm open to a future best C, which would be C89 + anonymous structs/union, and also accessing members of structures with incomplete types
10:13:25 * heat runs
10:14:23 <doug16k> incomplete types still work
10:15:14 <heat> IMO the best C is C89 + (C11 - C89)
10:15:17 <heat> basically C11
10:15:24 <heat> actually, gnu11
10:20:49 <zid> doug16k: You can't dereference them
10:21:35 <zid> so you can't do something like struct outer { int flag; struct opaque_inner o; } if you want to access 'flag' you have to know the declaration for o
10:21:50 <zid> Logically there shouldn't be a problem with doing this, though
10:22:13 <zid> it just needs to error when defined
10:25:10 <zid> Someone told me the plan9 'C' compiler actually supports it
10:33:48 <pterp> When init starts a bus driver(ex PCI), how does the bus driver know which device driver to start? File of class codes to driver names?
10:35:03 <bcos> I'd design the file names for matching
10:35:11 <pterp> how?
10:35:45 <bcos> (e.g. the file "/sys/drv/pci/v1234/d5678" would be the driver for PCI device with Vendor=1234 and deviceID=5678
10:35:52 <zid> pterp: linux modules provide an array of vendor:device pairs they support
10:36:33 <zid> windows has .inf files with the same
10:36:36 <bcos> zid: Seems backwards though - kernel would have to load the module to determine if it needs to load the module
10:36:41 <zid> bcos names his files by the vid:did
10:36:49 <zid> bcos: modules are pre-loaded to work
10:36:53 <zid> if they're not loaded they don't exist
10:37:00 <bcos> Eww
10:37:03 <zid> it's userspace's job to modprobe modules that are needed at runtime but not at boot
10:37:03 <heat_> personally I keep it simple and just load every module and driver and then it's their job to check if it supports any device or not
10:37:18 <zid> bcos: hotplug
10:37:39 <zid> You don't want 1GB of linux drivers in memory at all times
10:37:48 <zid> or even to map them all into memory and back to try running them
10:38:34 <zid> "Ignore the 20 second delay while I execute all 40k drivers I support to see if they claim any devices"
10:39:04 <heat_> doug16k: oh, actually I forgot, I wasn't using IRQs
10:39:43 <heat_> zid: right
10:39:58 <heat_> but I don't have 40k drivers so I haven't found the need to write something linux-esque
10:40:03 <zid> nod
10:40:12 <zid> I have erm, 2, drivers, so I have an if() :P
10:40:27 <heat_> :D
10:40:46 <heat_> Fuchsia uses bytecode iirc
10:41:13 <heat_> it's embedded in the drivers
10:43:33 <bcos> I like my idea better - try to find driver using file name based on vendorID/deviceID, then if not found try "class/subclass/progif", then try "class/subclass"
10:43:51 <zid> yea that's basically inf without the parsing step
10:43:57 <zid> but easier for the user to mess up :P
10:45:13 <bcos> No user
10:45:20 <doug16k> one might take issue with that exact scheme, but you'll probably end up with at least some variations on that scheme. using the filesystem for lookup is not a bad technique if the file counts are reasonable
10:45:42 <doug16k> you'll end up doing those lookups bcos mentioned one way or another though
10:46:15 <pterp> Ok. Different question. I've discovered the WEIRDEST bug in my paging code. There's a page in virtual memory that always points to a page directory (usually the current one, but can change when editing others). I have a statement that maps the 768th entry to the first kernel page table. It fails when run normally (Doesn't change anything), but works properly when done in the debugger. Any ideas on what's hapening?
10:46:44 <bcos> pterp: Do you invalidate stale TLB entries properly
10:46:45 <bcos> ?
10:46:49 <pterp> oh.
10:47:00 <bcos> :-)
10:47:28 <zid> "oh." :D
10:47:46 <pterp> What does that have anything to do with it though? It works when run in the debugger in place of the actuall compiled code?
10:48:17 <doug16k> in the debugger it is running incredibly slowly. any change will get picked right up. at full speed it's different
10:48:46 <doug16k> is this in tcg or bochs or kvm or bare machine or what?
10:48:48 <zid> paging code will all be simulateded by the vm/debugger/etc environment and may not even 'have' a tlb
10:49:15 <doug16k> zid, are you saying that it is going and doing a page walk every load and store? no
10:49:38 <doug16k> emulators need to cache translations even more than real machines do
10:49:42 <zid> no, I'm saying it's going to not bother as much as possible
10:49:56 <zid> and only expose things like the tlb to the emulated program when it's definitely looking at it
10:49:58 <zid> quantum tlb
10:50:24 <zid> so the behavior unless you *force* it to have one, with say, invlpg, is indeterminate, and probably whatever is fastest
10:50:42 <bcos> I remember one of my very first kernels. Written and tested on a Pentium CPU. Didn't bother doing any TLB invalidation at all; and it worked perfectly. Then Intel invented Pentium Pro and it was like my OS was bleeding out of every orifice.
10:50:51 <bcos> :-)
10:51:32 <zid> my OS just fails if you try to map the same page twice and has no munmap, eat it
10:51:37 <zid> better than bcos.com
10:51:48 <doug16k> it's hard enough to lookup a virtual TLB entry in an emulator, let alone making the TLB fully coherent
10:54:33 <heat_> doug16k: Ha! I fixed it! I forgot removing the install_irq and free_irq code from the pit code so apparently that was causing the emulator to freak out(probably because of the tinkering in the ioapic masks)
10:54:56 <pterp> Still doesn't work.
10:55:01 <pterp> :(
10:55:02 <doug16k> every load and store doing linear-to-virt-to-phys translations are the main bottleneck in an emulator
10:55:36 <doug16k> x86 one I mean
10:55:47 <nikuga9> heya
10:56:16 <doug16k> if you look at the die you see a giant amount of space used to make load/store as amazing as possible
10:58:51 <doug16k> it would be so fun to go back in time and sit in at meetings at intel. be that guy that keeps saying, "yeah but, can't they see which cache lines got pulled in speculatively?" and see how long it takes for it to click :D
10:59:44 <doug16k> they knew
10:59:48 <doug16k> the whole time
11:00:01 <nikuga9> would anybody here join an osdev discord with an actually sane adminship
11:00:12 <doug16k> probably not
11:00:12 <nikuga9> its a huge bummer that the only one is run by kinda crazy people
11:00:16 <heat_> no why would anyone do that
11:00:36 <jjuran> We have an osdev IRC channel with sane adminship
11:00:45 <heat_> discord for friends + games, irc for programming and nerd stuff
11:00:48 <doug16k> what is better in discord other than taking 3 orders of magnitude more resources to run a client?
11:01:02 <jjuran> It's proprietary?
11:01:19 <nikuga9> thats fair
11:01:32 <heat_> I like discord.
11:01:40 <nikuga9> how do modern emulators optimize virtual memory emulation anyway
11:01:56 <nikuga9> address translation and such
11:02:03 <heat_> It doesn't try to compete with irc
11:07:24 <doug16k> nikuga9, same as cpu. some limited number of virtual address translations are remembered and it will go page walk if it can't find it
11:07:32 <nikuga9> so with a virtual tlb?
11:07:43 <doug16k> yes
11:07:59 <doug16k> how else right?
11:08:06 <nikuga9> i guess thats convenient since you need to emulate a tlb anyway in some cases
11:08:08 <doug16k> page walk every load/store? that'd be crazy
11:09:38 <doug16k> remember that you do more than just find a physical address, you also have permissions to combine together
11:13:03 <doug16k> it's likely to be very small though to be fast enough to beat the full page walk significantly. being small makes it almost invisible as far as tlb bugs go