channel logs for 2004 - 2010 are archived at http://tunes.org/~nef/logs/old/ ·· can't be searched
#osdev2 = #osdev @ Libera from 23may2021 to present
#osdev @ OPN/FreeNode from 3apr2001 to 23may2021
all other channels are on OPN/FreeNode from 2004 to present
http://bespin.org/~qz/search/?view=1&c=osdev&y=19&m=2&d=2
12:30:38 <kingoffrance> rereading what you said knebulae, it almost sounds like you want process groups using one scheduler, other processes using another, all under same "master" kernel (i dont know enough to know what you mean by higher level page table)
12:30:45 <kingoffrance> almost like nested kernels, but only for select things
12:43:34 <nyc> It's probably more about supporting policy than it is mechanisms.
12:56:35 <knebulae> @nyc: bingo
12:57:03 <knebulae> but if the policy can be implicit in the design you don't actually have to write any code for it to be so.
12:57:25 <knebulae> (well not *no* code, lol)
12:57:49 <nyc> Once the policy is implicit in the design it's become a design flaw to everything whose needed policy conflicts.
12:58:17 <knebulae> @nyc: which is why I want many cores and "dynamically" adjustable policy among them.
12:58:34 <knebulae> but I hate that word in this context.
01:00:06 <nyc> (Hot-?)pluggable scheduler architectures are probably worth looking at e.g. Solaris.
01:02:07 <knebulae> @nyc: I'm also very interested in going further into Niall Douglas' research into blurring the lines between user and system allocators.
01:05:48 <bcos_> Fortunately, user-space allocators are completely in user-space, so it's easy to research alternative and/or special purpose allocator designs
01:06:43 <nyc> knebulae: Allocators of what?
01:07:04 <bcos_> "heap" would be the most likely
01:10:07 <knebulae> @bcos_: Niall states that there are additional performance gains that can be made by exposing limited kernel functionality to userspace (partially exposed usermode paging or even raw usermode paging).
01:10:24 <nyc> knebulae: My health insurance / healthcare needs make gig affairs non-starters for me. Maybe a kernel with aggressive design goals might raise enough eyebrows.
01:10:54 <nyc> knebulae: Asynchronous page faults? Userspace-driven page replacement?
01:11:07 <knebulae> I think if I can rig up everything on these sleds we can see massive parallel gains.
01:11:20 <knebulae> With less actual work for the kernel
01:12:05 <knebulae> @nyc: speaking my language
01:12:21 <knebulae> it all just flows. like it should. it's effortless.
01:12:23 <nyc> knebulae: I've never even had direct access to anything with CPU counts higher than 32x.
01:13:15 <knebulae> @nyc: only at AWS. :)
01:13:40 <knebulae> @nyc: but it's too costly.
01:14:32 <nyc> knebulae: There are ways to get people's attentions.
01:14:54 <nyc> knebulae: But they need some major kernel design features.
01:15:14 <knebulae> Well, I'm 26 days in. I figured having something that didn't sh*t the bed would be a good start.
01:15:20 <nyc> knebulae: Things would probably also have to run major databases out of the box.
01:16:36 <bcos_> knebulae: Too tired to read the paper properly; but after a breif glance it looks like Niall has been taking crack cocain intravenously.
01:16:42 <knebulae> @nyc: well, it doesn't even relocate itself yet. :/ lol.
01:17:08 <ryoshu> hmm is multiboot for amd64 too?
01:17:58 <knebulae> @bcos_: I hope you're talking about the rest of his stuff and not his memory allocation stuff. If so, maybe I need to re-think that little idea. Lol.
01:17:59 <doug16k> ryoshu, not exactly. it enters your stuff in 32 bit protected mode and you have to do things
01:18:46 <ryoshu> so can I boot a 64-bit kernel with multiboot?
01:18:53 <nyc> The major things I think the big iron vendors want are:
01:18:53 <nyc> 1. asynchronous file IO
01:18:53 <nyc> 2. pagetable sharing
01:18:53 <nyc> 3. transparent large pages for file & anon
01:18:53 <nyc> 4. M:N threading and other massive thread count support
01:19:29 <doug16k> ryoshu, you can, you just need a trampoline to setup paging and do the last steps for entering long mode then enter your kernel
01:20:03 <ryoshu> is it done by somebody?
01:20:05 <ryoshu> Linux?
01:20:10 <knebulae> @nyc: if that's actually true, then maybe I wasn't crazy all those years ago.
01:20:49 <knebulae> the hardware just wasn't there
01:20:53 <nyc> knebulae: With those things and enough ABI compatibility with something they already run on they'll probably grant hardware access.
01:21:39 <knebulae> @nyc: that's something that I had never, ever considered. I don't know that I'm a talented enough programmer to put code in front of people like that.
01:21:46 <nyc> knebulae: Well, it's been there since the early 80's, but at varying prices and floorspace requirements.
01:21:53 <bcos_> knebulae: I'm talking about a paper called "User Mode Memory Page Allocation", that presumes virtual memory isn't needed then fails to do things like benchmark the performance of file IO (once copy on write and file caches are gone) and only does micro-benchmarks that seem deliberately chosen to highlight the benefits and avoid the downsides
01:24:18 <nyc> Usually people start with e.g. TPC-C and do things specifically to optimize for that.
01:24:34 <ryoshu> https://wiki.osdev.org/Creating_a_64-bit_kernel
01:25:28 <knebulae> @bcos_: so shit. ok.
01:25:37 <knebulae> sorry for swearing.
01:30:15 <nyc``> I think just getting back to the point of what VMS used to br able to do in the 80's within the FOSS ecosphere would make a lot of people happy.
01:33:40 <knebulae> @nyc: which would be? Sorry, I've never used VMS.
01:34:34 <knebulae> Well, not from Digital anyway.
01:37:53 <doug16k> yay! rewrote kernel-module loader to use program headers and .dynamic and never touch a section header once
01:38:01 <doug16k> I did it the extremely hard way before :(
01:38:42 <klys> doug16k, are your kernel modules ELF
01:38:55 <doug16k> yes
01:39:08 <klys> are they in .o form
01:39:41 <doug16k> no. the kernel has a lot of exports (memory, pci, irqs, string, etc etc) and the modules dynamically link against the kernel exports
01:40:44 <doug16k> essentially shared libraries
01:40:50 <klys> so they're executable and rely on those exports with a sort of linker-loader?
01:41:02 <doug16k> right
01:42:29 <klys> do you have a current feature-list?
01:42:31 <nyc> doug16k: Which part is a DSO? The module or the kernel?
01:43:35 <doug16k> module. kernel is regular executable with -export-dynamic -fvisibility=hidden -fvisibility-inlines-hidden, and an export.h with #define EXPORT __attribute__((__visibility__("default")))
01:43:47 <doug16k> I can throw EXPORT before a function definition and it becomes exported
01:44:58 <nyc> knebulae: VMS mostly had clustering things beyond basic aio and shared pagetables.
01:46:33 <knebulae> @nyc: so if there's such a big clamoring for async i/o, why has that not been brought to linux?
01:47:46 <knebulae> forgive me- while I do administer some linux boxen, I am not very educated about the inner workings of the kernel.
01:48:16 <knebulae> I have heard people b*tch about it here and there (like on reddit)
01:48:24 <klys> knebulae, have you done "make menuconfig"
01:48:33 <knebulae> long, long time ago
01:48:38 <klys> ah
01:48:40 <nyc> knebulae: It was in some early Red Hat. Ben LaHaise is probably set for life after doing it all those years back.
01:49:16 <knebulae> it was never mainlined?
01:49:29 <doug16k> my import checker script -> https://gist.github.com/doug65536/ba4d6468bebb8cd3217462f6cb7c43a5
01:49:39 <doug16k> tells you things modules use that kernel isn't exporting (yet)
01:50:04 <nyc> knebulae: Not even close, no.
01:50:33 <nyc> knebulae: The stuff that actually got merged into mainline is grossly inferior, basically a fixed-size pool of in-kernel worker threads.
01:50:57 <knebulae> @nyc: I do think I remember reading about this. /.
01:52:03 <nyc> knebulae: It doesn't work well with UNIX-style vfs.
01:55:52 <knebulae> @nyc: doesn't async just cause UNIX-style workflows to stutter a bit?
01:56:43 <knebulae> @nyc: but the delay shouldn't exceed any delay that sync code would have also experienced.
01:57:04 <knebulae> I mean with proper libc support.
01:57:05 <nyc> knebulae: I don't know. Usually the database box doesn't run anything else.
01:57:40 <knebulae> @nyc: right, which is why they don't care what's good for UNIX workflows.
01:58:47 <nyc> Usually the relevant disks / spindles to the database are irrelevant to interactive users and everything but the database.
01:58:49 <knebulae> @nyc: kind of why I'm surprised there hasn't been an exokernel that's broken through. The big DBs are the prime candidates.
01:59:42 <knebulae> @nyc: there's folks out there doing what looks cool (at least from afar)
02:00:25 <nyc> knebulae: I think the databases wouldn't actually interfere with normal UNIX processes etc.
02:01:43 <nyc> knebulae: They use different disks, avoid in-kernel buffering and run with their own in-userspace buffer management, are mostly IO bound, etc.
02:03:15 <knebulae> @nyc: I see.
02:08:47 <knebulae> @nyc: a comment on HN circa 2016 pointed me in the right direction. Linux devs argue (maybe rightfully so) that the same mechanics can be accomplished via sync i/o on a separate thread. Certainly not eloquent, but not wrong.
02:09:32 <nyc> knebulae: I disagree.
02:10:06 <knebulae> @nyc: ok
02:10:39 <knebulae> You'd still be in a polling situation though (as per the comment on HN).
02:11:25 <knebulae> Not nearly as nice as being jolted back into the runqueue the second your i/o request has been serviced.
02:13:47 <nyc> knebulae: The semantics of waiting for any one of a pool of events to finish in any order is very different.
02:15:07 <knebulae> @nyc: right
02:21:07 <doug16k> klys, you mean my feature list? yeah my readme does slip behind, I forget to add big additions there
02:21:46 <klys> doug16k, well it would be nice;
02:30:24 <knebulae> @nyc: well, it hasn't really proven to be a dealbreaker on linux, where big commercial DBs are available.
02:30:53 <knebulae> They must internalize the complexity of juggling their own in-app version of async-io.
02:31:04 <knebulae> Which seems like a giant PITA tbh.
02:31:06 <nyc> knebulae: They are not happy with it.
02:31:54 <knebulae> Lotsa wasted cpu
02:38:23 <kingoffrance> only tangentially related, some ppl swear the way to "profile" is simply pause a program many times and see what function it is inside (versus building with profiling etc.) not something that can be done on production things per se, but an os/kernel that had a built-in standard way to do that (or kept tally of such things), rather than "by hand", might be useful
02:38:46 <kingoffrance> "poor mans profiler"
02:40:11 <chrisf> kingoffrance: this is more common than you think.
02:41:05 <knebulae> @kingoffrance: on POSIX, SIGTRAP?
02:41:14 <chrisf> kingoffrance: sampling profilers are very real and useful
02:41:33 <knebulae> I mean like a SIGTRAP button?
02:41:46 <knebulae> or icon?
02:42:44 <kingoffrance> the ppl who do that i think basically just script around gdb
02:43:09 <geist> aren't we all, after all, just scripts around gdb?
02:44:49 <nyc> Sampling profiling is standard.
02:44:56 <nyc> oprofile used to do that on Linux.
02:45:09 <kingoffrance> knebulae: i just meant like an env. var or something, or even ld_preload library or something
02:45:35 <kingoffrance> or even touch something in /proc so you can enable/disable at runtime
02:47:23 <nyc> I don't know the full array of issues involved. I think it wants to dump virtual memory layout like what shared libraries are where and so on.
02:48:18 <nyc> Then with minimal information collection at interrupt-time it can batch process samples later and trace through libraries, executables, kernel, and sources.
02:49:49 <vendu> hello world :)
02:49:55 <geist> hola vendo
02:49:58 <geist> vendu even, dang!
02:50:00 <vendu> yo geist :)
02:50:02 <vendu> hehe
02:50:03 <vendu> :)
02:50:22 <vendu> geist, try spelling this: epäjärjestelmällistyttämättömyydellänsäkään :)
02:50:33 <geist> is that that place in Wales?
02:50:34 <vendu> rest assured it's a word in my language i.e. finnish :)
02:50:43 <geist> ah finnish, equally odd language
02:51:34 <geist> https://www.youtube.com/watch?v=dCGkqUr1kbY that guy nails it and loves it
02:52:06 <vendu> hmm :)
02:53:33 <geist> or the icelandic name for their big volcano
02:54:58 <nyc> Eyjafjallajokull?
02:55:04 <kingoffrance> geist: well, if computer originally is "one who computes", then people are the original debuggers too. soylent gdb is made of people.
02:55:48 <nyc> http://gen.lib.rus.ec/book/index.php?md5=5FCC863A50F12AA08E9DEB85BF0CC8A5
02:57:02 <geist> nyc: yeah that one
02:57:33 <Ameisen> "epäjärjestelmällistyttämättömyydellänsäkään"
02:57:35 <Ameisen> did I spell it right?
02:58:21 <Ameisen> Finnish is like that because the language doesn't separate what Indo-European languages treat as particles, conjugations, declensions, and modifiers.
02:58:24 <Ameisen> They just get added onto the word.
02:58:52 <geist> yah but this one is supposed to be the longest non compound word
02:59:04 <geist> since yeah compounds you can probably go on and on
02:59:06 <Ameisen> silly agglutinative languages like all the Uralic languages
02:59:39 <vendu> https://github.com/vendu/valhalla/blob/master/v0/ARCHITECTURE.txt
02:59:40 <vendu> :D
02:59:48 <vendu> comments/feedback, anyone?
02:59:51 <geist> oh no that's a compatar
03:00:03 <vendu> geist, a plan for something to run a synth + console with :)
03:00:06 <vendu> retro style console
03:00:09 <Ameisen> Unlike the masterful fusional languages.
03:00:17 <Ameisen> or English, which is somewhere between fusional and analytic...
03:00:21 <geist> vendu: suggestion: draw a map of the bits with ascii art
03:00:23 <geist> that really helps
03:00:24 <vendu> but the processor is going to have proper elements of multithread/core/processor/user stuff :)
03:00:37 <vendu> geist, good idea dude, thanks :)
03:00:48 <vendu> geist, i've been using LaTeX for some stuff so far :)
03:00:57 <geist> even something like https://github.com/travisg/2stage/blob/master/isa.txt#L37 i thnk helps a bit
03:01:08 <geist> doesnt' have to be ascii art as a way to visualize the opcode layout
03:01:18 <vendu> geist, bit patterns :)
03:01:22 <vendu> geist, more compact
03:01:23 <vendu> hmm
03:01:42 <vendu> geist, the grouping seems relatively logical but the actual pipelines i probably need to fine-tune :)
03:01:44 <geist> i think you start to see more patterns when you write them ot that way too
03:01:52 <vendu> geist, i drew great inspiration from the cray 1 :)
03:01:57 <vendu> some from risc-v
03:02:03 <vendu> and other architectures (arm, mips, ...)
03:02:19 <vendu> geist, simd i have intentionally kept out so far :P
03:02:26 <vendu> need to figure that stuff out one day :P
03:02:28 <geist> yah
03:02:31 <vendu> but i'm sorta low on resources
03:02:43 <vendu> geist, http://vendu.twodots.nl/files/papilio_pro-arcade.jpg :)
03:04:09 <geist> oh very cute
03:04:22 <vendu> geist, less than 100 bucks to get :)
03:04:28 <vendu> hold on a sec =)
03:05:10 <vendu> http://store.gadgetfactory.net/arcade-megawing-papilio-bundle/
03:05:16 <vendu> i have the papilio pro fpga board :)
03:05:44 <vendu> geist, i have plans and ideas for a computer game, too
03:05:45 <geist> yeah that's pretty cute
03:05:51 <vendu> kinda like nethack/rogue meet cyberpunk :)
03:06:13 <vendu> geist, familiar with core war? :)
03:06:20 <geist> yes
03:06:34 <vendu> geist, i have an implementation of the 1989-ish instruction set :)
03:06:42 <vendu> probably to be used as a subgame in mjolnir :)
03:06:44 <vendu> the game :P
03:06:55 <geist> years ago a friend of mine tossed together an ARM based corewars thing
03:06:56 <vendu> geist, let me show you ;)
03:06:59 <geist> each side got a cycle, etc
03:07:07 <vendu> :)
03:07:13 <vendu> geist, i have it done in c :)
03:07:18 <geist> it was pretty neat. you could submit a program and it would run a match like 10000 times and see who comes out ahead
03:07:22 <vendu> hehe
03:07:24 <vendu> yeah :)
03:07:55 <vendu> geist, i think the dungeon will have punchcards (fixed core war warriors), tapes (stored ones), terminals and mainframes :)
03:08:09 <vendu> so you could gain the mastership of the mainframes by beating the warrior currently owning it :)
03:08:28 <vendu> so the game, especially if online, would get gradually harder as the gamers get better :)
03:08:43 <vendu> geist, core war <3
03:09:19 <vendu> https://github.com/vendu/OS-Zero/blob/master/usr/games/mjolnir/IDEAS.txt
03:10:14 <geist> neat
03:12:34 <vendu> geist, are you aware you can teach your dog/pet to shoplift in nethack? :)
03:12:51 <vendu> so it's possible to five-finger discount stuffs :)
03:14:02 <knebulae> @nyc: one last thought: with proper semantic separation of the kernel (think namespaces or jails), with my idea, it should be possible to allow a trusted application (such as a large DB) to run in supervisor mode, to completely eliminate context switching, if you properly mask the interrupts, disallow the initiation of any IPI, and disallow changing CR3. Maybe?
03:16:27 <knebulae> If you box it in through the vmmap, you should be able to keep it under control.
03:18:52 <nyc> knebulae: They'll never go for that.
03:20:41 <knebulae> @nyc: I know. But I like to play fast & loose sometimes.
03:23:29 <knebulae> Dangle 30% performance gains in front of them with the caveat that the DB undergo double-top-secret validation?
03:24:28 <knebulae> I'm kidding. Later guys.
04:09:58 <knebulae> @bcos_: you were correct. I actually read the first page of Niall's paper, and you're absolutely correct; he goes off the rails pretty quickly. Honestly, I had followed the trail from dlmalloc, who recommended nedmalloc as a multithreaded extension. I only reviewed the blurb on his website about usermode page tables, which jives with something that I've been pondering, namely that you could allow "true" usermode multitasking if a runtime could
04:09:58 <knebulae> manipulate its own pagetables, instead of just stack switching.
04:13:10 <knebulae> I haven't done anything with his allocator because I haven't implemented any synchronization primitives; now I'm not so sure I want to go near his code.
04:22:10 <kingoffrance> you havent done web dev nyc, some places just chmod a+rwx everything to "make things work"
04:22:36 <kingoffrance> i totally see "so it'll run faster as root/etc. ?"
04:23:00 <kingoffrance> even less braindead people, just disable selinux to "make it work"
04:30:04 <doug16k> asked in #gcc, maybe someone here has encountered this: why is this explicit instantiation .hidden -> https://godbolt.org/z/SIRLbs
04:30:40 <doug16k> I'm doing the instantiations in other files already that work, so I'm really puzzled
04:32:17 <nyc> kingoffrance: You're right, I haven't done webdev. I don't have the artistic inclination etc.
04:35:19 <doug16k> it _must_ be in a namespace?! -> https://godbolt.org/z/OGKteq
04:37:14 <knebulae> @doug16k: to prevent naming clashes?
04:38:20 <knebulae> nvm.
04:43:12 <klys> doug16k can you call F::F() explicitly from a separate module? or does the symbol not appear in nm -s ?
04:43:42 <doug16k> klys, I can tell it won't work from another module because of the .hidden _ZN1FIiEC1Ev
04:43:55 <klys> yeah that's true
04:44:05 <doug16k> the G one (in a namespace) isn't .hidden
04:45:11 <klys> well if you don't define F::F() in that same module, instead defining it in a separate module, then that's different, I suppose.
04:45:47 <klys> not too clear on why that would be.
04:46:58 <kingoffrance> nyc: ive seen one case of someone using selinux. they needed auditing like their prior OS, so it was just to log things.
04:48:57 <doug16k> klys, the header would say `extern template class F<int>;` That means "I swear there is an instantiation of F<int> somewhere the linker can find, don't bother generating that class with those types from that template, assume you can just refer to the symbols"
04:49:51 <doug16k> in the cc file, it would say `template class F<int>;` That means, "please spit out the implementation of F with T=int right here so other things can just link against it"
04:50:13 <doug16k> I'm trying to make that spat out thing also be visibility default (exported)
04:51:01 <klys> is there a reason you're defining the function inside the class body, then?
04:51:31 <doug16k> that's just a minimal example that strips away all my specifics
04:51:34 <klys> I guess you could just declare it there and define it right afterwards...
04:51:48 <doug16k> that constructor is .hidden. point made
04:52:16 <doug16k> ah you mean maybe it is inlined? hmm let me see
04:53:00 <doug16k> nope, still .hidden -> https://godbolt.org/z/SaNwnX
04:54:10 <doug16k> to be sure -> https://godbolt.org/z/5Xg--p
04:56:12 <knebulae> @doug16k: so I decided to try and research your issue. I stumbled across this article: https://gcc.gnu.org/wiki/Visibility -- written by none other than Niall Douglass (see earlier posts). FML. I've gone full circle..
04:57:27 <doug16k> knebulae, related, but doesn't cover explicit instantiation and doesn't even contain the word namespace
04:57:46 <knebulae> I know.
04:58:01 <knebulae> I didn't say the article actually answered any questions.
04:58:18 <doug16k> I guess I should just put those in a namespace and call it a day
04:58:33 <doug16k> I usually like to figure out why when this happens though
04:59:05 <doug16k> ah... see last paragraph. https://gcc.gnu.org/onlinedocs/gcc-6.3.0/gcc/Visibility-Pragmas.html
04:59:07 <doug16k> strange
04:59:26 <doug16k> you can't put attributes on explicit instantiations, at all
04:59:40 <doug16k> there's a special warning to bark about that specifically
08:34:59 <nyc`> I got so waylaid that I don't even remember where I was stuck.
08:35:30 <nyc`> I suspect it had something to do with MIPS firmware on qemu.
08:43:40 * geist reads about APL for some reason
08:46:32 <nyc`> APL was worse than perl for write-only code.
08:54:03 <klys> geist, http://www.rexswain.com/aplkeyb.gif
08:58:50 <geist> yah it was because i was reading about an IBM5100
08:58:55 <geist> el psy congroo
09:00:08 <klys> apl is sort of a logic language, with actual logic symbols, i gather
09:00:29 <geist> it does some mafs too
09:01:29 <klys> math
09:01:47 <nyc`> I think it's imperative with vectorized operations everywhere.
09:03:50 <nyc`> (I use actual logic languages and APL was never one. Prolog is the premier example. I use Mercury, which is like a cross between Prolog and Haskell.)
09:09:03 <geist> yah i was joking about prolog a few years ago, thinking that no one used it anymore
09:09:12 <geist> turns out it's still got some use
09:14:36 <zhiayang> peculiar question: how does one begin to debug a situation where one thread is inexplicably running slower than another?
09:14:57 <zhiayang> the asm of each is virtually identical to the other (excluding some string offsets & jump addresses)
09:15:21 <bcos_> How do you measure "running slower"?
09:15:42 <zhiayang> i'm printing some chars at intervals
09:16:01 <zhiayang> so like while(true) { if(++ctr % N == 0) print('x'); }
09:16:33 <zhiayang> so the metric is "prints less chars"
09:16:44 <zhiayang> (before being preempted)
09:16:45 <bcos_> So, could be that the "slower" thread is running faster but the scheduler is giving it less CPU time?
09:17:25 <bcos_> ..or it's being interrupted by IRQs more
09:17:54 <zhiayang> there's literally nothing else going on, so i would think both would have an equal probablity of getting irqed
09:18:18 <geist> is it printing chars to a screen or uart or something?
09:18:24 <bcos_> ..or differences in CPU speed (power management, turbo-boost, what the other logical CPU in the same core is doing)
09:18:44 <geist> what's the environment it'sr running on?
09:18:50 <zhiayang> i've restricted it to serial in case '2' takes longer to draw than '1'
09:19:01 <zhiayang> qemu
09:19:12 <geist> on your os?
09:19:16 <zhiayang> yep
09:19:27 <geist> does it spin while trying to print to the uart?
09:19:37 <geist> as in what would cause it to context switch?
09:19:41 <zhiayang> there's no locks anywhere
09:19:46 <zhiayang> fixed interval pit
09:20:04 <geist> ah, dunno then. there's nothing based on this description that would cause what you say
09:20:15 <zhiayang> i'm perplexed as well
09:20:38 <bcos_> What's the fixed interval?
09:21:14 <renopt> what's the difference in the number of characters printed by each?
09:21:20 <geist> if they're truly identical code then i dunno
09:21:38 <geist> otherwise i'd check to see if either thread does something that would cause it to vmexit
09:21:51 <geist> those tend to take an enormous ammount of time that appears hidden
09:22:14 <zhiayang> vmexit?
09:22:19 <geist> assuming you're using KVM. if you'r enot then its not as profound an effect, but doing io can cause the emulator take a long time
09:22:29 <bcos_> (Qemu tends to emulate instructions in "bursts of N ms", so if the scheduler's fixed interval is clost to that length you might get "lumpy timing")
09:22:35 <geist> so what seems harmess (reading or writing a register or io port) may take essentially thousands of cycles
09:22:43 <zhiayang> hmm
09:23:13 <geist> oh that's true too, actually. if the inner loop of one of the threads has a large block it may context switch at a different spot
09:23:39 <geist> what bcos_ said. TCG is a trace based jit, so it tends to treat a large block of code as a single unit until it hits a branch or whatnot
09:23:44 <zhiayang> i've set the divisor to 2^16, and i'm switching every 200 interrupts
09:24:05 <geist> it only really checks for IRQ testing at block boundaries
09:24:10 <zhiayang> hm
09:24:29 <geist> though it'll probably be skewed by the tracing, may be fascinating to look at it via -d cpu,exec
09:24:45 <geist> which will print a firehose of info, but there's a chance if you stare at it something will stick out
09:25:58 <geist> also are you emulating SMP and running more than one core?
09:26:40 <bcos_> D'oh - caches? If both threads use same data, does slow thread get cache misses and then fast thread uses the already cached data?
09:26:53 <zhiayang> geist: nope, no smp going on
09:27:05 <geist> bcos_: probably not with TCG. caching effects are probably nonexistant there
09:27:09 <bcos_> (that could include "JITted instruction cache" maybe)
09:27:34 <bcos_> ..and "host CPU's cache"
09:27:43 <geist> good, SMP and TCG also has weird effects as qemu essentially round robins between emulated cpus
09:28:17 <geist> zhiayang: i'd start by looking at precisely what the difference is between the code stream of the two threads
09:28:34 <geist> maybe start by literally running the same code in two threads, so that they're identical except for stack
09:28:57 <geist> i bet you'll find that one of them is doing something different that may cause the emulation to take a lot longer
09:29:20 <zhiayang> i'll get an updated disassembly
09:29:37 <geist> you said the asm is virtually identical, that's not exactly identical, so worth starting there
09:29:38 <zhiayang> but, the code is basically identical. strangely if i make them exactly identical, there's no problem
09:29:46 <zhiayang> but if i change a "?" to a "!"
09:29:49 <zhiayang> then boof
09:29:53 <geist> then there you go
09:30:00 <geist> so now look at what that does
09:30:06 <geist> to the generated code
09:32:00 <zhiayang> the asm is really identical save the expected data/code addresses
09:32:19 <zhiayang> and the instructions are the same length as well
09:32:46 <zhiayang> https://bpaste.net/show/186255e9a07a and https://bpaste.net/show/886e5e4cd1ba
09:33:00 <geist> fascinating
09:33:24 <geist> which one is slower?
09:33:38 <zhiayang> work_thread2
09:34:19 <geist> only difference i see there is it crosses a page boundary
09:34:22 <geist> but that shouldn't matter at all
09:35:04 <geist> but what is this ? vs !?
09:35:16 <geist> what did that change do to the codegen?
09:36:23 <zhiayang> string literal
09:37:09 <geist> then i have no idea whats going on
09:37:25 <geist> it's clearly something else
09:37:34 <geist> but given what you have said there should be no reason for it
09:37:34 <zhiayang> gg
09:37:54 <geist> which means it's some other thing you haven't noticed yet
09:38:26 <renopt> the ! startles the CPU
09:40:20 <zhiayang> the initial call to println (outside the loop) appears to do something
09:40:31 <zhiayang> in that if a function is called i get the strange behaviour, if not then no
09:40:44 <zhiayang> in that if a function is called i get the strange behaviour, if not then no
09:40:46 <zhiayang> oops
09:41:21 <geist> could be the page crosses causes TCG to jit the code slightly differently
09:41:45 <geist> and normally you dont notice this sort of thing, except when you finely 'benchmark' the code like you're essentially doing
09:41:49 <zhiayang> the thing is, is this just some harmless nonsense i can safely ignore
09:42:07 <zhiayang> or is it one of those things that will collapse everything when i least expect it
09:42:12 <geist> most likely. the closer you try to get down to deep timing details and whatot in emulators, the more you will be completely surprised
09:42:22 <geist> it's not reality, it's a simulation of something that acts roughly like a computer
09:42:39 <geist> in aggregate it seems to work the same way, but the closer you zoom in the more it gets wonky
09:42:44 <zhiayang> hm, alright then
09:42:52 <zhiayang> i'll keep a record of this somewhere
09:42:54 <zhiayang> just in case
09:43:12 <geist> it's also possible this is a canary for something more fundamental
09:43:22 <geist> so if other things start acting wonky, then be very suspicious
09:43:36 <zhiayang> will do
09:50:19 <renopt> so, I've been trying to get my shiny new ioapic code working on real hardware, but no luck, I'm now stumped
09:51:42 <renopt> PIC is disabled, spiv software enable is enabled, pretty sure I'm mapping MP I/O interrupt entries correctly, apic TPR is 0, apic timer is working, interrupts enabled on all CPUs...
09:52:08 <renopt> what else should I be looking at?
09:52:57 <renopt> (have some crummy debug logs here, if it's worth anything => https://p.sicp.me/pjSme.txt)
09:54:33 <doug16k> renopt, does the real hardware have more than one ioapic? it might
09:54:57 <doug16k> my dev machine has two
09:55:23 <renopt> only one ioapic specified in the MP tables, but the driver should handle multiple correctly too
09:57:19 <doug16k> can you put in code that reads the entire (256 bit) ISR from the LAPIC? if any bits are set you're stuck on an unacknowledged IRQ
09:58:28 <doug16k> (the set bit(s) indicate unacknowledged pending IRQs)
09:58:38 <nyc`> geist: Rumor has it the IBM Watson AI uses Prolog.
09:58:42 <doug16k> where "acknowledged" means "did an EOI"
09:58:58 <renopt> oic, worth a shot
09:59:28 <geist> renopt: if you're using qemu or soemthing i highly recommend tracing through its code and adding printfs to see whats causing it not to do what you want
09:59:41 <geist> building it from scratch so you can add printfs is extremely useful
10:00:11 <nyc`> What's TCG?
10:00:40 <geist> that's qemu's software emulation mode
10:00:46 <geist> trace code generator or something like that
10:00:55 <geist> it's the name for that engine
10:00:56 <klange> It's a sort of bytecode interpreter.
10:01:02 <nyc`> oic thanks
10:01:04 <doug16k> renopt, I'd dump both the ISR and the IRR while you're at it
10:02:11 <klange> guest → TCG → host, sometimes with guest == host, but with all the protection aspects of a software VM; quite speedy compared to traditional direct emulators, you can actually use modern stuff under it without stabbing your eyes out...
10:02:40 <geist> yah, it's apparently 'tiny code generator'
10:03:20 <geist> i've looked at it's implementation, not half bad. basically translates emulated code to an intermediate, and then converts that to host instructions
10:03:30 <geist> it builds up little traces of single runs of code up until a branch or whatnot
10:04:35 <bcos_> renopt: A single IO APIC with 8 redirection table is abnormal (need a total of at least 16 to cover ISA IRQs, and typically there's a total of 20 or more). Either it's multiple IO APICs or there's more rediction table entries; which means you're either parsing the MP tables wrong or the MP tables are wrong. For modern computers the latter is likely (MP is deprecated in favour of ACPI and MP tables can be "bare minimum stubs")
10:05:10 <geist> bcos_: not sure about that. i think it only needs as many redir entries as there are legacy irqs
10:05:44 <renopt> bcos_: hmm, there's 23 redirection entries, the ioapic ID is 8, the logs are formatted like garbage I know (sorry heh)
10:05:47 <bcos_> Sure - there's 15 legacy IRQs so it only needs 15 and 8 isn't enough
10:05:56 <nyc`> I really need to get PCI scanning for serial started on ARM and SPARC even if MIPS is lacking firmware to invoke to get a starting point. I couldn't make heads or tails of what Linux did without firmware from reading the source alone, but probably could go on a fishing expedition for it tomorrow with debuggers or some such.
10:06:04 <bcos_> renopt: Ah - that makes sense :-)
10:06:28 <geist> nyc`: ARM it generally just uses the device tree where the PCI bits are
10:06:44 <geist> or hard coded because the SoC specific driver Just Knows
10:06:54 <geist> likely to be how it works in most MIPS or other machines
10:07:24 <bcos_> renopt: For your logs, what does "trigger mode: 0" mean?
10:08:06 <renopt> level/edge triggered, iirc 1 is level, 0 is edge
10:08:20 <nyc`> geist: qemu docs didn't say and Linux source didn't give too many hints. ARM I think has EFI to use on qemu.
10:08:43 <geist> nyc`: zircon already supports it. it's in the FDT
10:09:10 <geist> it says where the PCI ECAM is, or you can simply Just Know, because it's at a fixed address
10:09:34 <bcos_> renopt: Usually there's "edge/level" and "active high/low"; but MP spec (and ACPI) have this annoying "zero means it conforms to the standard for the bus" crap where you need extra code to figure out what the bits could've (and should've) been; and without that extra code (if you just take the bits literally) you mis-configure the trigger
10:09:35 <geist> with -machine=virt that is
10:09:46 <zhiayang> geist: out of curiosity i swapped the code positions (so the other function crosses the page boundary) -- i'm getting the same results, but flipped
10:09:50 <geist> with other emulated machines that have PCI, it's anyones guess, but usually it's FDT
10:09:53 <zhiayang> (ie. the function that crosses the boundary is "slower")
10:09:59 <geist> zhiayang: interesitng.
10:10:13 <nyc`> geist: FDT it is, then.
10:10:14 <geist> may be that it causes it to break a TCG run and thats enough to make it a tiny bit slower
10:10:26 <zhiayang> perhaps
10:10:28 <geist> nyc`: or you just know. never underestimate how useful that actually is
10:10:35 <zhiayang> hopefully this is indication that it's not something more sinister
10:10:38 <geist> it's how a large portion of the !PC world works
10:10:40 <zhiayang> so i'll just leave it
10:10:50 <renopt> bcos_: ooooh, true, hmm
10:11:18 <geist> as in you use some indicator that you're on a particular SoC. either via hard compiling the code to run on it, or by having some arg passed to the kernel in some wya or oanother that says 'you're running on a XYZ'
10:11:32 <geist> and then you just have tables of where everything is and start bringing up drivers that way
10:11:37 <geist> it's not pretty, but it actually works
10:12:08 <geist> or some combination of having a dynamic structure that describes where things are and some amount of Just KNowing
10:12:18 <geist> ie, a FDT that has some amount of info
10:12:43 <nyc`> Firmware needs magic addresses to be invoked, so it's just how much indirection there is.
10:12:53 <geist> there is no real standard way of doing this, though in general ARM linux has moved towards FDT
10:13:14 <geist> note that the presence of firmware is also not a given
10:13:37 <geist> when booting qemu arm for example, unless you tell it it usually has no ROM at all
10:13:51 <geist> or at least -machine=virt -kernel=<your elf file>
10:13:56 <nyc`> Oh sure, things can be the first invoked.
10:14:05 <geist> it simply blats it in memory and passes as an arg to it a constructed FDT
10:14:21 <geist> as if there had been some sort of loader that had done it
10:14:32 <doug16k> when you get the index on the stack when a plt stub is called and lazily links it, what is that index? got index? can't be though, got 0 thru 2 are taken. do I add 3 to that and index into PLTGOT and set that GOT entry?
10:15:22 <doug16k> I mean $index1 in this example -> https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter6-1235.html#feowg
10:16:31 <doug16k> hmmm. I think I can infer it from the initial state. I'll just see which got entry (index) is pointing back to that push and reverse figure it out
10:22:52 <doug16k> lol, really? -> https://gist.github.com/doug65536/cc7ab411624252c47b197fa810896a3c
10:35:21 <renopt> doug16k: so, nothing in the ISR/IRRs
10:38:22 <renopt> bcos_: active low/edge triggered should work for the ISA bus anyway right, I'm just trying to get the keyboard working at least lol
10:39:03 <renopt> I'll add proper bus-dependent configuration later
10:39:26 <bcos_> renopt: I can't remember if ISA was active low or active high
10:50:54 <doug16k> ISA should be active high, edge, and PCI should be active low, level, IIRC
10:51:21 <doug16k> when it gives that goofy "default" (0) that bcos mentioned
10:52:42 <doug16k> I remember because PCI can share, so therefore must be level, and the 4 interrupts are INTA# INTB# ... INTD# (the # means active low)
10:56:17 <renopt> man
10:56:21 <renopt> that's it isn't it
10:58:57 <bcos_> It might make things worse; but for edge triggered "active high" really means "on rising edge", and device would send a pulse (rising edge, then falling edge) so (in theory) if you get it wrong you'd still get an IRQ (later than you should - at end of pulse instead of at start of pulse)
10:59:32 <bcos_> ..but who knows what other shenanigans are between device and IO APIC
11:00:45 <renopt> and indeed, that was it
11:00:58 <bcos_> Cool :-)
11:01:04 <doug16k> nice
11:01:06 <renopt> thanks bcos_, doug16k, I spent way too much time on that holy turds
11:07:07 <doug16k> it makes sense for shared IRQs to be level triggered active low. the motherboard holds the INTx# line high through a resistor. when the device wants to assert an interrupt, it pulls it low to ground. if two or more hold it to ground, no problem. when you ack the first IRQ on the device, the interrupt line will stay asserted by the other device. when you ack the 2nd one it will really deassert the INTx# line
11:09:18 <SopaXorzTaker> i.e. wired-OR
11:09:26 <doug16k> yes
11:15:02 <Jenz> Hey, anyone know of a tutorial/walkthrough/resource on building any kind of kernel for the esp32
11:15:05 <Jenz> -?
11:19:27 <Jenz> (esp32 is a SoC with a Tensilica Xtensa LX6 µ-processor)
11:38:02 * Jenz is very patient
11:38:27 <Jenz> And moin to you all!
12:05:37 <doug16k> does this look about right? https://gist.github.com/doug65536/0030fbf3f6e710c2e7c9d102de0fa12e
12:08:05 <doug16k> it works btw
12:13:59 <doug16k> normally I'd have DT_BIND_NOW in there and resolve all jump slots at load time, but I did the lazy bind capability for the hell of it
12:18:08 <kingoffrance> re: gcc visibility link earlier "ELF's exported symbol table format is quite a space hog, giving the complete mangled symbol name which with heavy template usage can average around 1000 bytes." im guessing that is because the mangling needs to be "reversable", else theyd just use a hash or similar
12:18:53 <doug16k> maybe a hog for a giant library like libstdc++ yeah
12:21:57 <doug16k> you don't necessarily touch a lot of the pages though, depends on how many different things you call and how they are distributed in the symbol table
12:51:37 <Jenz> tbh, I don't get the xtensa lx6 architechture at all
12:51:48 <Jenz> It's weird
01:13:29 <nyc`> C++ in-kernel worries me.
01:21:18 <nyc`> I sit around thinking about the per-user-thread overhead of kernel stacks and scaling from 1MB to 1PB RAM seamlessly. I feel like my grip on things is lost in C++. But I really want templates bad. Maybe Ada generics are the way.
01:22:03 <klange> i mostly sit around thinking about retiring
01:23:01 <nyc`> klange: Must be nice if it's a serious possibility for you.
01:23:52 <klange> from osdev :P
01:24:12 <klange> though I do also regularly think about what targets I need to hit to retire from everything else, too
01:24:43 <klange> get the right job, pay off a mortgage, build a better nestegg... maybe try to earn an income through some alternative means to full time employment...
01:24:54 <klange> unfortunately no one wants to pay into a patreon for a hobby OS
01:26:31 <nyc`> klange: What does it offer them that other OS's don't?
01:27:27 <klange> nothing, and anyone who thinks they have anything to offer over linux's incredible corporate backing is kidding them selves
01:27:45 <klange> but people pay into patreons for all sorts of shit that offers them nothing
01:30:27 <nyc`> klange: Heh, the corporate sorts were pulling their hair out about how uncooperative etc. the community was when my career was still active.
01:31:40 <nyc`> klange: There are plenty of things that ate great that are basically either permavetoed or can't be retrofitted onto such a rapidly churning codebase.
01:32:20 <klange> Which doesn't matter in the slightest as no lone developer is going to shit out something better for enterprise. The target moves too rapidly.
01:33:06 <nyc`> klange: Witness the 80's VMS technology still with no FOSS equivalent, for example.
01:44:29 <nyc> klange: I think the featureset the enterprise is looking for out of the core kernel is actually doable by a lone kernel hacker.
01:45:43 <nyc> The tough part is getting access to higher-end hardware to port it and to performance tune it.
01:46:11 <gedeysenegulderi> Hey
01:46:19 <nyc> And, of course, drivers.
01:46:34 <nyc> gedeysenegulderi: Hallo!
01:46:41 <gedeysenegulderi> Do I need to manually init the stack when writing a bootloader on x86 real mode?
01:47:25 <nyc> gedeysenegulderi: I think so though I couldn't prove it with documentation.
01:47:51 <gedeysenegulderi> So I can just set SP to something and thats it?
01:48:28 * Jenz o/
01:49:04 <nyc> gedeysenegulderi: I think care needs to be taken for it to point to actual memory and also for it to avoid the BIOS.
01:50:26 <gedeysenegulderi> If my CALLs and RETs work, does that mean than my stack works too?
01:50:31 <gedeysenegulderi> Cuz they work without any init
01:51:04 <klange> The stack is *somewhere* when the BIOS passes control to you.
01:51:12 <klange> But it's probably not where you want it, and it may wipe out your other data.
02:03:38 <nyc> Amoeba and Sprite are both open source and did SSI load-balancing clustering, Sprite even with process migration.
02:04:19 <gedeysenegulderi> What are they
02:04:48 <nyc> gedeysenegulderi: Operating systems!
02:04:58 <nyc> Sprite: https://www2.eecs.berkeley.edu/Research/Projects/CS/sprite/sprite.html
02:05:21 <nyc> Amoeba: https://www.cs.vu.nl/pub/amoeba/
02:07:06 <gedeysenegulderi> wow first-world universities making OSes while ones in my country make propaganda pamplets denying evolution
02:07:40 <nyc> gedeysenegulderi: :( I'm sorry that's happening to you.
02:09:14 <gedeysenegulderi> you should be, i can't even get a tourist visa to see the outside world
02:10:18 <klange> Where are you from?
02:10:38 <gedeysenegulderi> azerbaijan
02:11:06 <gedeysenegulderi> north korea lite basically, instead of exporting fear we export oil
02:12:45 <nyc> https://en.wikipedia.org/wiki/Multikernel <-- someone was mentioning this earlier
02:12:51 <klange> You can visit... Russia, Malaysia, the Ukraine... not a lot of great options for visa-free.
02:14:10 <gedeysenegulderi> No thanks
02:14:23 <klange> I hear Malaysia's actually pretty nice.
02:14:58 <gedeysenegulderi> I don't like fattening criminal regimes with my money
02:15:36 <nyc> Ugh, time to close the laptop and run off to an appointment.
02:16:05 <gedeysenegulderi> Unlike your people who don't give a single shit about staying in hotels owned by the dictator himself in my country
02:16:35 <gedeysenegulderi> Basically any 5 star one
02:17:29 <klange> Not sure who you think my "people" are.
02:18:08 <gedeysenegulderi> Basically anyone who doesn't need to collect 200 documents to apply for a tourist visa to anywhere sane and get denied
02:19:14 <gedeysenegulderi> Or people who has been within 100 miles of an university that can even think of making an OS, even a Hello World one
02:19:51 <gedeysenegulderi> They just don't give a shit
02:20:47 <nyc`> Sorry things are so bad in your country. Hopefully kernel programming will uplift you somehow.
02:21:14 <gedeysenegulderi> Doubt it
02:26:05 <gedeysenegulderi> all you fuckers ever do is offer platitudes
02:26:15 <gedeysenegulderi> thoughts & prayers
02:26:21 <nyc`> I think gang scheduling is pretty important esp. for virtualization.
02:28:03 <nyc`> gedeysenegulderi: I got tossed out of the industry on my arse 10 years ago and have been in rather dire poverty ever since. I can't help you get remotee work.
02:28:25 <gedeysenegulderi> Why were you tossed out of the industry? What does that even mean?
02:29:02 <nyc`> gedeysenegulderi: Fired and blacklisted so no one will ever hire me again.
02:29:18 <gedeysenegulderi> Blacklisted where?
02:29:46 <gedeysenegulderi> You were banned from the whole computing industry?
02:30:11 <nyc`> gedeysenegulderi: Yes.
02:30:22 <gedeysenegulderi> Do you live in a small country or something?
02:30:44 <nyc`> gedeysenegulderi: The US.
02:30:54 <gedeysenegulderi> I don't see how you could get blacklisted unless you are talking about criminal record
02:31:05 <nyc`> Really, gang scheduling is a wonderful thing.
02:31:24 <klange> nyc has something worse than a criminal record in the eyes of some people.
02:31:44 <gedeysenegulderi> Like what?
02:32:09 <gedeysenegulderi> Sex abuser list?
02:32:21 <gedeysenegulderi> offender*
02:32:22 <nyc`> One might even say that gang scheduling is a critical enterprise feature.
02:32:58 <nyc`> gedeysenegulderi: No.
02:33:20 <gedeysenegulderi> Unless you got caught spying for China I don't see what the fuck you are talking about
02:33:41 <gedeysenegulderi> In which case you probably deserve what you got
02:34:59 <nyc`> Critical enterprise features are helpful to attract attention, funding, etc. to OS's.
02:35:33 <klange> gedeysenegulderi: I've about had enough of your language.
02:35:55 <nyc`> I'm foggy on how Linux attracted those things without having those features first.
02:36:28 <gedeysenegulderi> Having servers that actually can run stuff is pretty enterprise feature if you ask me
02:38:04 <nyc`> And I can't imagine the vendors are satisfied even now.
02:42:04 <gedeysenegulderi> I was spared
02:43:51 <klange> Only because it's nearly midnight in my timezone.
02:44:17 <gedeysenegulderi> Use a silencer
02:47:39 <nyc`> The state of kernel aio really has to bring them to tears. I wonder why shpte ever made mainline and am amazed that Oracle is still carrying it as a patch. Superpages are a wreck and are still happening to a lesser degree than internal IBM patches were implementing in 2003. It's all really sad. This is all preexisting OS technology, not even new inventions, and things shipped in commercial OS's at that.
02:54:23 <nyc`> Hugh and Sunil even have a war story about how SCO SVR4 got ported to some Pentium Pro SMP box and they ended up catching a race in hardware pagetable walking triggered by the pagetable sharing that it already had.
02:56:55 <gedeysenegulderi> How do you even get to know about such crazy OS history trivia?
02:57:28 <nyc`> Yeah, shpte is 80's technology.
02:59:23 <gedeysenegulderi> What is shpte?
02:59:41 <nyc`> I'm telling it secondhand. Hugh Dickins and Sunil Saxena were the actual people involved.
03:00:18 <nyc`> shpte is pagetable sharing or shares pagetables.
03:00:36 <nyc`> Shared pagetables.
03:02:10 <nyc`> It's common for separate full processes to attach to large shared memory segments while using private memory mappings etc.
03:03:57 <nyc`> Basically like thread-local storage on steroids. They can also maintain smaller-scope shared memory segments. It goes on.
03:04:42 <gedeysenegulderi> Can you eat it though?
03:05:37 <nyc`> The point is that these large numbers of full processes are actually doing something reasonable.
03:06:35 <nyc`> It's not just a forkbomb loop.
03:08:39 <gedeysenegulderi> Forkbomb is like cutting edge tech in my country
03:10:13 <nyc`> They're not using threads because they need local address space.
03:14:57 <nyc`> There are
03:15:32 <zhiayang> anyone with experience dealing with virtualbox's acpi?
03:15:38 <zhiayang> specifically their MADT seems a bit whack
03:15:56 <zhiayang> the field that's supposed to be the ioapic's id appears to correspond instead to the number of processors on the system
03:16:14 <zhiayang> (reading the ioapic's own registers shows that the real id is 0)
03:16:34 <zhiayang> (also the rest of the fields seem legit, because the base address gives me a usable ioapic config -- so it's just the one 'id' field)
03:17:34 <knebulae> @nyc: you mentioned C++ and then Ada; I think you're thought process is following where my head has been.
03:18:33 <kingoffrance> well http://www.groklaw.net/articlebasic.php?story=20060626215210721 sco 7.1 had a "freeware license" if someone is curious how they did async io. i got a vm and built cross gcc/binutils for it.
03:21:06 <knebulae> @zhiayang: that has not been my experience with VirtualBox acpi
03:22:07 <knebulae> VirtualBox lapics are numbers 1...n. You don't actually get a cpu count; you just count the entries.
03:22:28 <zhiayang> ioapics, not lapic
03:22:43 <zhiayang> and yea i'm not expecting it to be a cpu count, but in practice that's what i'm observing
03:22:51 <zhiayang> i set 5 cpus, the id is 5; i set 1, it's 1; etc. etc
03:27:13 <knebulae> @zhiayang: sorry, misread; just woke up.
03:27:31 <knebulae> @zhiayang: they might do that just for convenience
03:27:40 <zhiayang> well that's just bogus
03:28:08 <zhiayang> i'll dig in their source
03:28:17 <knebulae> @zhiayang: you ain't seen nothing yet; between qemu and VirtualBox, it gets interesting.
03:29:02 <zhiayang> rip
03:32:12 <zhiayang> hey what the fuck
03:32:13 <zhiayang> https://www.virtualbox.org/browser/vbox/trunk/src/VBox/Devices/PC/DevACPI.cpp#L2993
03:32:14 <zhiayang> i knew it
03:32:17 <zhiayang> wankers
03:42:45 <olsner> "@todo All hardcoded, should set this up based on the actual VM config!!!!!" :)
03:43:35 <olsner> makes sense to me though - each cpu gets a lapic id 0..n-1 then the ioapic gets the next apic id which is n
03:48:29 <zhiayang> ya, if they're gonna do that then they should set the IOAPIC's register value accordingly!
03:58:12 <nyc`> knebulae: You mean templates / parametric polymorphism?
03:59:51 <knebulae> @zhiayang: as expected
04:00:09 <zhiayang> i'm more upset than i feel i should be
04:00:13 <zhiayang> but i am quite upset
04:00:22 <zhiayang> for now i'll just make it a warning instead of an abort
04:00:43 <nyc`> knebulae: The pain comes when trying to do an in-core B+ tree with C++ templates or Ada generics.
04:01:02 <zhiayang> the question is how are other oses not affected by this? do they not tally the madt info with the actual ioapic info
04:01:11 <knebulae> @nyc: blows the cache
04:01:18 <nyc`> Or any sort of genericity whatsoever.
04:02:03 <nyc`> knebulae: The point is to have cache and TLB locality of reference.
04:02:17 <knebulae> @zhiayang: I don't know that the reported id of the ioapic is germane to its use.
04:02:28 <knebulae> @zhiayang: just that it has *an* id
04:03:42 <nyc`> It probably matters more when you have a dozen of them.
04:03:57 <zhiayang> right, i don't see myself caring too much about their ids either
04:04:16 <zhiayang> but still
04:17:40 <nyc`> knebulae: I think the ID gets used somewhere to identify the source of the interrupt or otherwise to index into the memory range to acknowledge the interrupt. I'm not where I can look at the docs just yet.
04:21:15 <nyc`> knebulae: I weirdly can't /msg you.
04:22:25 <zhiayang> from what i can see you don't actually need to know the ioapic's id, because they should have a unique base address
04:22:38 <zhiayang> and it's kinda set-and-forget, since interrupts get eoi-ed on the lapic
04:24:38 <nyc`> There is something relevant somewhere. Maybe hotplugging devices.
04:28:00 <nyc`> I haven't touched a NUMA-Q since 2003. All of the IO-APIC hassles had to do with the gaggles of IO-APIC's on those things.
04:37:52 <bcos_> Maybe the "directed EOI" part of x2APIC (hard to remember)
04:53:06 <kingoffrance> re linux async io https://lwn.net/Articles/724198/ thats kinda sad too "That makes it impossible to add a new flag, since a calling program can never know whether the kernel it is running on supports that flag or not." even a noob like me, knows how to duck type
04:54:09 <nyc`> kingoffrance: What happened to suparna?
04:54:26 <kingoffrance> <-- noob . i have no idea what suparna is
04:54:55 <kingoffrance> the closest i have got to async io was serial port library i wanted such a thing, on freebsd
04:56:16 <nyc`> kingoffrance: suparna was the dev IBM had parked on aio.
04:57:19 <nyc`> kingoffrance: She wrote a fair amount of what's in there today.
04:59:49 <bcos_> kingoffrance: It's a lot easier to shrug and tell yourself "Linux is shit"..
05:00:18 <nyc`> kingoffrance: I think RHAS2.1 had bcrl's true aio.
05:03:09 <nyc`> kingoffrance: bcrl == Ben LaHaise
05:08:49 <kingoffrance> well, even my total noob case, it is either peg cpu 100%, or insert arbitrary sleeps. so even single process slowly interfacing to serial port at 9600, some type of async is useful
05:08:59 <kingoffrance> i just thought arbitrary sleeps were ugly
05:12:18 <bcos_> kingoffrance: It's not supposed to be hard - you shove the data into a buffer and "prime the pump" (send the first byte/s) if necessary, then return so the caller can do whatever it likes without blocking. Then when you get "transmitter empty" IRQs from serial port you shove more data (from buffer) at it. Eventually it finishes and you either send some kind of "async IO finished" notification back to caller, or set a flag for caller to poll (and have scheduler
05:12:20 <bcos_> do a "block this task until flag set" thing if caller wants to wait until it's finished)
05:14:45 <nyc> kingoffrance: aio is really an issue for bdevs ... cdevs are another matter despite technically being async.
05:15:33 <nyc> kingoffrance: It's really an artifact of the UNIX VFS design.
05:20:04 <nyc> It's actually tough to find RHAS2.1 kernel code.
05:30:36 <nyc> kingoffrance: http://ftp.redhat.com/pub/redhat/linux/enterprise/2.1AS/en/os/i386/SRPMS/
05:52:24 <kingoffrance> wrong person nyc? or were you showing me something?
05:53:05 <nyc> kingoffrance: You were talking about aio.
05:55:17 <kingoffrance> i did learn something else, my serial port thing was: a | b | c | a so that is tricky with bash, circular pipes. i think i had to use bash "coprocess", but rc shell is one line
05:55:54 <kingoffrance> does anyone really need such things? i just thought it was fun
05:58:46 <FireFly> I think you could do that with Unix pipes?
05:59:22 <FireFly> mkfifo loop; a <loop | b | c >loop # or so
06:15:25 <geist> good morning folks
06:17:19 <doug16k> nyc, there's no significant stack difference in C++
06:24:18 <nyc> doug16k: I was trying to convey that I'm obsessive about cutting down overheads (e.g. those stacks) that are in some respects small in comparison to C++ runtime requirements.
06:24:43 <geist> well, it's a real thing in kernel space, since stacks are usually perma-wired
06:25:26 <geist> but if you're comparing C++ to C the stack usage shouldn't really go up per se, except that generally c++ generates More Code
06:25:36 <geist> so you perhaps will get bigger and or more functions
06:25:52 <geist> but that's fairly subjective. depends a lot on the style of the code
06:26:20 <nyc> I didn't mean to imply a comparison between stack usages between the two.
06:26:59 <geist> yah i didn't bother to read the backtrace. too much of a wall of text last night
06:31:43 <geist> that being said, you should be worried about c++ in the kernel, but i can tell you it's completely doable
06:31:56 <geist> it just requires some careful consideration, especially of object lifecycle
06:32:12 <geist> and hidden allocations
06:33:55 <kingoffrance> FireFly: yeah, i had it working on a few shells/mechanisms, i mean without using mkfifo :)
06:35:48 <nyc> Well, B+ trees do show some promising locality of reference effects at https://panthema.net/2007/stx-btree/speedtest/
06:35:54 <nyc> And they use C++ templates.
06:39:20 <geist> right
06:39:30 <geist> you have to keep an eye on things, but if you do it's doable
06:40:34 <geist> my expereicen is that c++ tends to generate a lot of code, however, so i wouldn't use it on a tightly embedded system necessarily
06:48:59 <kingoffrance> FireFly: i checked my backups. on windows i just piped to netcat, on unix-ish i did use mkfifo, but i coulda swore i had an es shell version
06:54:52 <kingoffrance> FireFly: the other thing i had to do was foo with stdout piped to pgma, stderr piped to pgmb, so maybe that is what i am misremembering
09:12:58 <kingoffrance> https://defuse.ca/b/dGtSt4sN i should probably stop calling that "duck typing", but i dont know another name except "try not to break API". it just seems common sense to me, for any library/API.
09:15:22 <mrvn> That has nothing to do with duck typing.
09:16:03 <mrvn> don't use "duck" for that.
09:16:51 <kingoffrance> what should i call it? ducked argument values...if the library/object says it supports a feature, assume it does
09:16:58 <mrvn> It's not a duck
09:17:19 <mrvn> it's just a way to have a variable numnber of features.
09:18:03 <kingoffrance> i know, but thats not a catchy name
09:18:41 <CompanionCube> so what?
09:18:51 <CompanionCube> not having a catchy name is better than misusing another term
09:19:05 <mrvn> A ducked function would accept a void * and expect that to point to something that behaves like a duck even if it is a dog. E.g. Have a make_a_noise() function pointer as first member of a struct.
09:20:00 <mrvn> and still it's not right because no typing happens.
09:21:12 <geist> yah that's basically objective c
09:21:33 <geist> at least for methods
09:21:45 <geist> dont think you can dynamically look up fields in a struct
09:22:25 <bcos_> For the original problem; I'd be tempted to split the "reserved" flags in half - a group of "reserved hints" (that kernel ignores if they're set) and a group of "reserved features" (that kernel returns error if set)
09:23:02 <bcos_> 'course that doesn't help much when it'd too late and unknown/reserved flags are already being ignored
09:25:00 <bcos_> Also doesn't really help much when you confuse "kernel API" with "user-space library interface"
09:25:20 <bcos_> (e.g. you can't just change the lib when behaviour of kernel API changes)
09:26:37 <mrvn> geist: nah, you need to hash the function name to get an offset or something.
09:27:19 <mrvn> You have features that must be set, can be set, must not be set and can not be set. Also features that depend on others, subfeatures.
09:28:03 <mrvn> e.g. in ext4 the hashed_dir feature must be accepted by the reader if the FS has it. It breaks compatibility.
09:28:35 <bcos_> You can halve that by choosing wording in the description better (e.g. "disable foo, must be set" can just as easily be "enable foo, must be clear")
09:29:04 <mrvn> or features that disappear if not met by the user.
09:29:06 <geist> ext* have a set of feature flags that are kind of neat
09:29:14 <geist> they have i think the hard features, RO features and RW features
09:29:26 <geist> tht way, for example, ifyou dont know how dir_hash works you can still RO mount it, because that's a RW feature
09:29:45 <CompanionCube> it's not just an ext4 thing
09:29:48 <CompanionCube> openzfs has them too
09:29:55 <mrvn> yeah. iirc hashed_dir can be read but not written without but can be cleared at that time.
09:29:56 <geist> yah it's a nice way to deal with that sort of thing
09:30:09 <geist> mrvn: or you can ignore it, since i think the dir still is linked
09:30:16 <geist> you just dont get the fast lookup
09:30:22 <CompanionCube> (the ZFS ones also have a notion of 'enabled-but-not-active')
09:30:26 <mrvn> geist: That's what I mend by "can be read"
09:31:00 <mrvn> features and backwards compatibility is the hardest to design I feel.
09:31:04 <geist> right
09:31:15 <mrvn> anbd maintain
09:31:38 <kingoffrance> itd say its somewhat opposite of duck typing: probe first, dont just invoke a function. "function feature probing"
09:31:50 <CompanionCube> I think they also recently grew 'types' even
09:39:31 <CompanionCube> yep, just found it
10:33:46 <klys> .theo
10:33:46 <glenda> You are the type of people who create these situations.
11:06:07 <Plazma> .theo
11:06:07 <glenda> Totall irrelevant.
11:44:57 <bluezinc> .theo
11:44:57 <glenda> Are you done?
11:59:16 <shikhin> .theo
11:59:16 <glenda> You are very confused.