@(#)Driver.porting.notes 1.2 92/04/08 NOTICE: This file is under construction. DO NOT PUBLICIZE ITS EXISTENCE without checking with Deborah Bennett (deborah@dobbs.eng) first. This file is considered Sun Proprietary. Do not disseminate the contents outside of Sun Microsystems. Notes on porting your 4.x device driver to SunOS/SVR4 5.0 (jupiter) =================================================================== Comments on this information should be directed to Deborah Bennett, deborah@eng. Please do not edit this file without asking first. Thanks. The notes in this file will eventually be compiled into a supplement to the manual "Writing Device Drivers" (hereafter referred to as WDD). There is an beta draft of the Jupiter WDD, part number 800-6502-06, dated February 1992. Or you can contact Kennan Rossi (kennan@eng) for any later documentation. Late news - the porting appendix *did not* make it into the Jupiter FCS documentation. Deborah Bennett (deborah@eng) is writing a white paper covering driver porting. She also has an 11-page presentation on drivers in Solaris 2.0 which can be given to customers. There is a mail alias for discussions of porting drivers to the sun DDI. The alias is ddi-owners@dgdo. Requests to be added should be sent to ddi-owners-request@dobbs.eng. Important Definitions: WDD: Writing Device Drivers manual Bootstick: The new boot/prom/kernel interface system. devfs: The new way of creating the links in /dev so you can access your device Jupiter: Solaris 2.0 or SunOS 5.0. The first customer release of SVR4 on Sun platforms Zeus: a previous (developer's) release. Sun4c only. Mars: a future 5.0 dot-release. The cutover release. This guide has two parts: - Overview of changes which affect driver writers. This includes some basic reasoning behind the Sun DDI/DKI, new boot-stick, and devfs systems. - Step by step instructions, organized by driver entry points, on what needs changing in your driver. If you find info is missing, PLEASE send mail to deborah@eng so that info can be put in this doc. ################################################################# ## Overview of Changes in Jupiter which affect Driver Writers ## ################################################################# **** What is the Sun DDI/DKI? **** (References: gigo:/export/nse/docs/i-ddi/ddidki/* pretty badly out of date Jupiter WDD - 800-6502-** Section 9 manual pages on a 5.0 system near you - latest manpages are on plummet:/usr/src/man) Why Invent a Sun DDI/DKI: - The DDI/DKI is a new name for the routines formally called "kernel support routines" in the old WDD, and for the "well-known" entry points in the cdevsw/bdevsw structs. The intent is to specify a set of interfaces for drivers which can specify a binary and source code interface specification for device drivers. If your driver only uses routines and structures described in the section 9 manpages (DDI-compliant), then your driver is guaranteed to be binary compatible across Sun Solaris platforms with the same processor (like sun4c, sun4, sun4m, sun4d). - Sun extended the AT&T DDI/DKI to support multiple architectures, loadable drivers and modules, and many other things. **** What's new about boot? **** (Reference: gigo:/export/nse/docs/i-sun4c/jupiterboot.ps somewhat out of date, but still valuable) **** What's new this new devfs? **** (References: - Device Driver Installation, Configuration and the Physical Namespace by Tom Allen, Bill Pittore and Joe Provino - 5.0 Boot and Configuration by Joe Provino latest versions of both on wizard:/callisto/docs/misc/techs) **** ANSI C **** - Use "volatile" and "const" type specifiers where appropriate. Prototype all entry points. **** Header Files **** See Include Files below. The headers have been greatly reorganized. **** Things you can't do anymore **** - Many architecture-specific features have been hidden from driver writers behind DDI interfaces. Specific examples are elements of the dev_info struct, caches, memory management units (mmu), page tables. If you have been using unadvertised interfaces, you may have to re-architect your driver to be DDI-compliant. If you continue to use unadvertised interfaces, you lose all the source and binary compatibility guarantees of the DDI. - You can no longer access the user-process area corresponding to your driver thread of control. After all, it could be running on a different processor on an MP machine. No poking around in the user struct or proc struct. **** New Features **** - most of kernel and all drivers are now loadable. ###################################################################### ## Step by Step instructions on what needs changing in your driver ## ###################################################################### Below, I describe a general strategy to convert your driver from 4.x to 5.0. It is not the only way. The strategy is to convert as much as possible under 4.x before ever compiling on 5.0. (Loadablility falls here). Then you convert enough to get the driver to compile on the 5.0 system. Then you comment out all the guts of all entry points except those dealing with loading and unloading (_init, _fini, xx_identify, xx_probe, and xx_attach). Getting the driver to load and unload is at least half the total work involved for most drivers. Once it loads and unloads, then you can debug each entry point in turn until you're done. Steps in converting your driver to SunOS/SVR4 5.0 (jupiter). Step 0) Make it loadable. ------------------------- If your current driver is not loadable, I suggest that you convert your driver to a loadable driver (if you can) under 4.x before starting the port to the DDI. This will familiarize you with the loadable driver concept. The loadable functionality under Jupiter is not that different from that under 4.x on the Sun4c. There is good information on loadable drivers under 4.x in the manual "Writing SBus Device Drivers" (p/n 800-5323-05, part of the SBus developer's kit). Step 1) Get the driver to compile under 5.0. ------------------------------------------- Inside of Sun Engineering, There are two ways to do this. Either you can compile your driver using the installed SPARCcompilers (I call this building native) or you can activate an NSE environment read-only to use your local execset compilers. By this point (after beta) you should build native unless you are working on a driver which is tracking a special environment for some reason (such as it's for a new architecture which isn't netinstallable yet). Outside of Sun Engineering, building native is your only choice. **** MAKEFILE NOTES: **** Building Native: To compile the driver native on your jupiter system (jup_beta and later) - Use these compile lines: cc -D_KERNEL -I. -c example_one.c cc -D_KERNEL -I. -c example_two.c as -P -D_ASM -D_KERNEL -I. -o example_asm.o example_asm.s ld -r -o example example_one.o example_two.o example_asm.o NOTE: unlike under 4.x, you MUST do an "ld" step, even if your driver has only one .o file. You cannot modload a .o file anymore. Building under a read-only NSE activation: - Pick the relevant NSE environment. If you don't know which one you need to use, you probably should be building native. (see above) - activate your desired environment read-only: activate -r - cd to the directory where your source code is kept. Note that this directory should be outside of an NSE control point (that is, not under /proto or /usr/src). Do a "make" as usual. **** INCLUDE FILES **** All the kernel header files are located in the sys directory now. New header files you will certainly need are: # Note that these two headers MUST appear last. # in the kernel header includes. - the file has changed name to . - For full DDI compliance, you are only allowed to include kernel header files which are listed in the synopsis sections of the section 9 manual pages. No other kernel header files are allowed. **** STRUCTURE CHANGES **** - Add or change your xx_ops structure to be the new form. See the dev_ops(9s) manpage. If you have no devo_reset routine, use nulldev. - Define an xx_cb_ops structure. See the cb_ops(9s) manpage. If you have no devmap routine, use nodev. If you have no detach routine, use nulldev. Be sure to set the cb_flag field to ( D_NEW | D_MTSAFE ) if your driver is MT-safe. - Eliminate your local cdevsw or bdevsw structure. - The struct vdldrv has changed name to struct modldrv, and its form has also changed. See the modldrv(9s) manpage. Be sure to declare mod_driverops or mod_pseudodrvops as extern structs as well. - Add the new struct modlinkage. See the modlinkage(9s) manpage. - If you used to use the mb structures, they are gone. The elements of the mb_driver struct are (more or less) contained in the dev_ops struct. The elements which used to be in the mb_ctlr and mb_device structures are now opaque data in the dev_info struct. The information in your xx.conf file will cause some properties to be created for your driver when it is loaded. **** OTHER COMPILE NOTES **** - If under 4.1 you used to use the construct #ident ""; <--- note trailing semicolon you must drop the semicolon now. If you were still using static char SCCSid[] = ""; you should change that to the #ident form, with no semicolon. - the type "addr_t" is gone, replaced with "caddr_t". - Just to get the driver to compile, you can stub out and just prototype the new routines needed in the dev_ops and cb_ops structures, such as xx_getinfo. You will need to write these routines before the driver has any chance of loading, though. **** GOOD PRACTICES **** Prototype all your entry points in the ansi-c style as noted in the WDD. Use the new "volatile" type specifier to declare software structs which overlay actual hardware registers, or for specifying any other location which could be updated by an alternate thread of control. Also, use the new "const" type specifier for an object or array which is initialized, but then never changed. See any good Ansi-C manual for further details. Step 2) Get the driver to load and unload under 5.0. ---------------------------------------------------- First a note about modload versus first-reference loading. In 4.x, the only way to load a loadable driver was with the modload utility. In 5.0, drivers can be automatically loaded when their corresponding node in /devices is referenced. However, during early driver porting, you may want to still use modload and modunload until you are fairly sure that your driver loads and unload properly. Then you can add your driver to the system using the add_drv utility. Note that modload now requires a full pathname. See the modload(1m) manpage. **** LOADABLE DRIVER SUPPORT **** These routines must be working for loading/unloading to work properly: _init _fini _info xx_identify xx_probe (for NSID devices only) xx_attach xx_detach (if your driver will be unloadable) xx_getinfo You should be able to stub out the rest of your driver routines while debugging the loading/unloading. Once the driver loads and unloads, most of the work will be done for simple drivers. - A preliminary note: the kernel log() function and kernel printf() have been replaced by cmn_err(). See the cmn_err(9f) manpage. Note that CE_PANIC will panic the system after printing your message, and that some levels of cmn_err generate automatic labels before your message, and newlines after. - The routine you might have had in 4.x called xx_init, or xx_vdcmd, which had three sections labelled VDLOAD, VDUNLOAD and VDSTAT, has now been (approximately only) replaced with the three entry points, _init, _fini, and _info. _init: (see the _fini(9e) manpage) - This routine should do any per-driver (rather than per-unit) initialization, such as allocating per-driver memory. I believe that this routine is called before xx_identify(), xx_probe(), and xx_attach(), so don't assume anything about unit numbers or unit state structures being present. Anything which used to be in the VDLOAD case of xx_init belongs here, except that you no longer need to call vd_installmod. - I suggest that you call ddi_soft_state_init here. _init must call mod_install with the address of your modlinkage struct. _fini: (see the _fini(9e) manpage) - This routine should do any per-driver shutdown. Basically it should do quiesce all the units controlled by the driver, then free any memory allocated in _init. This routine will be called just before the driver executable image is removed from memory. Anything which used to be in the VDUNLOAD case of init_module belongs here, except that you no longer need to call vd_removemod. If the decommissioning is sucessful, call mod_remove with your modlinkage struct. Otherwise return an appropriate errno. _info (see the _fini(9e) manpage) - This routine is passed a modinfop *. You should return the return value from mod_info thusly: return (mod_info(&modlinkage, modinfop)); You can also print out information about your driver. You may find the __TIME__ and __DATE__ preprocessor directives handy to record the compile date of the driver. xx_info or xx_getinfo (see the getinfo(9e) manpage) This new routine should return a dev_info pointer for the dev_t (DDI_INFO_DEVT2DEVINFO) or the instance (unit) corresponding to the dev_t (DDI_INFO_DEVT2INSTANCE). - xx_identify() (see the identify(9e) manpage) - The "name" is no longer passed to you, you must use ddi_get_name() to get it. - IMPORTANT: the unit number counting is now handled by the system framework. To get the unit number in any routine, call ddi_get_instance(dip). DO NOT COUNT UNITS ANYWHERE! Consider this another good reason to use the new ddi_soft_state_* routines to manage your state structures. (They're internally MT-safe, too.) - You can no longer count on xx_identify() being called for all units before xx_attach() is ever called. - Return value should be DDI_IDENTIFIED or DDI_NOT_IDENTIFIED. - xx_probe() (see the probe(9e) manpage) - Used to determine whether the NSID (non-self-identifying device) which might be present is present, and if so, what version, configuration, etc it is. - If you used to statically allocate an array of controller state structures and use them this way: register struct xx_ctlr *c = &xx_ctlrs[ctlr]; you can now do this: c = (struct xx_ctlr *) kmem_zalloc (sizeof (*c), KM_SLEEP); ddi_set_driver_private(dev, (caddr_t) c); You then use ddi_get_driver_private to get the ctlr pointer elsewhere in the driver. You should also store your xx_units structs in the dev_info corresponding to each unit in the equivalent way. - If your driver supports multiple types of devices, you should use ddi_prop_create(9f) to create descriptive properties which the other driver routines may use to determine what to do. You may create as many properties as appropriate for your device. Note that the information in your xx.conf file will be turned into properties in your dev_info struct by the module loading process automagically. - You must now use ddi_map_regs to map in your registers. - xx_attach() (see the attach(9e) manpage) - xx_attach arguments have changed. - As noted above under xx_identify, don't count units anywhere. Use ddi_get_instance(dip) to get your unit (instance) number. - kmem_*alloc() now require a second (flags) argument. See the kmem_alloc(9f) and kmem_zalloc(9f) manpages. - The slaveslot() routine has been replaced with ddi_slaveonly(9f). - Use ddi_map_regs() to map in your registers where you used to use map_regs() or mapin(). - If you need iopb memory, the call rmalloc(iopbmap, size) can be replaced with ddi_iopb_alloc(), then ddi_dma_addr_setup(). You may also need to call ddi_dma_devalign(). - You are no longer allowed to look at the dev_info struct contents. The dev_info_t is an opaque pointer. There are various DDI routines to acquire the property information which used to be accessed by looking in the dev_info. See the ddi_prop_create(9f) and ddi_prop_op(9f) manpages for more details on property management. - addintr is replaced by ddi_add_intr. See the ddi_add_intr(9f) manpage. - It is no longer necessary to notify the system that you are using DVMA with a call to something like adddma. - If you used to use the splN and splr routines, the multi- threading of the kernel now makes these routines obsolete. You should be using mutexes to block out threads of control. A full discussion of this is beyond the scope of a porting guide. See also the discussion on making your driver MT-safe, below. You should call mutex_init on your mutexes here in xx_attach. - SBus drivers which followed the example of the sample driver in the SBus WDD probably have an xx_poll() routine which determines which unit is interrupting, then calls xx_intr() with the unit number. This is no longer necessary. You can call ddi_add_intr(9f) for each instance, use xx_intr as the int_handler argument, and pass the unit number or pointer to the unit structure in the int_handler_arg argument. This is a more general approach, and should work without modification if your device ever exists on another bus, such as an autovectoring bus like VME. - You should call ddi_create_minor_node for each attach. This creates the minor node structure used by the devfs to make the nodes in /devices at boot -r time. Be sure to call ddi_remove_minor_node if the call fails. - xx_attach should be careful to undo any unit allocations if a unit fails to attach. Be very careful to watch for recursive mutex_enter calls here. - the peek and poke calls have been replace with ddi_peek and ddi_poke. There are ddi_peek* calls for all sizes. - You may have to re-architect your driver since the mb structures are gone. The info in your xx.conf file will be turned into properties in your dev_info structure by the module loading process. Use ddi_get*prop() to fetch these properties. xx_detach: (see the detach(9e) manpage) - This new entry point allows your driver to be automatically unloaded to save system resources. NOTE: It is up to the driver to >allow< unloading. If you say no, the system cannot forcibly unload your driver. - xx_detach should undo and deallocate all the actions and allocations done in xx_attach. Be very careful to watch for recursive mutex_enter calls here. - If you previously had a decommissioning subroutine which you called in the VDUNLOAD step of your init_module routine, you can probably rework it slightly and make it your xx_detach routine. **** DEBUGGING HINTS FOR LOADING/UNLOADING **** - You can vary the value of the system global "moddebug" to see what's happening at module loading/unloading time. The possible values are in . - Unlike in 4.x, you can now symbolically debug with kadb and loadable drivers. Remember that your kernel symbols will not be mapped into the kernel at all times, only when your driver is loaded. Step 3) Get the rest of the driver entry points to work under 5.0. - xx_open() (See the open(9e) manpage) - note that the first arg to open() ONLY is a dev_t *, not a dev_t. - it is good practice to verify that the "otyp" you are passed is OTYP_CHR for character devices, or OTYP_BLK for block devices. This avoids having your driver opened in a layered open unexpectedly. These defines are in . - You are guaranteed that the system will not call the open routine for a particular unit unless the identify, probe and attach routines for that unit have been called and have returned. But this is per-instance, not per-driver. - xx_close() (See the close(9e) manpage) - Note that reac() now has two more arguments. - xx_read() (See the open(9e) manpage) - If you used to check user privelages using suser(), you should now use driv_priv(credp) instead. - IMPORTANT: you can no longer call physio passing the address of a buffer you have statically allocated. You may pass "(struct buf *) 0" as the second argument, allowing physio to allocate a buffer from the pool. If you must change this, remember to squirrel away the address of the allocated buffer in xx_strategy, because you need it to call biodone in your interrupt routine. Another strategy is to use getrbuf(9f) to allocate the buf struct, and freerbuf(9f) to free it. - If you are a DVMA device, and your dma engine has limited bits of resolution (such as the top 8 bits being registered rather than a counter), then you should set up a ddi_dma_lim_t structure describing these limits. Pass the address of this struct as the 5th argument to ddi_dma_buf_setup(). See ddi_dma_lim(9s). - If you need to DVMA data larger than MAXPHYS, see the ddi_dma_movwin(9f) manpage for details on the new sliding DVMA window feature. - The sleep() and wakeup() functions are still around, but should not be called from an MT-safe driver. Use cv_wait(9f), cv_signal(9f) and cv_broadcast(9f) instead. - xx_write() (See the write(9e) manpage) - Same notes as for xx_read, above. - xx_ioctl() (See the ioctl(9e) manpage) - Note that ioctl() now has two more arguments. - If you used to define your ioctl cmd args this way: #define MYIOCTL _IOR(m, 1, u_int) <-- m is not in quotes you will have to surround the letter m with single quotes to get the #define to work. Or you can use the new definition method: (left-shifted letter or-ed with number) #define DIOC ('d'<<8) #define DKIOCGGEOM (DIOC|2) Theoretically, the magic number you use (DIOC above) should be allocated by AT&T. The cred_p can be used to check credentials on the call, and the rval_p can be used to return a pointer to a return value which means something (as opposed to the old method of always getting zero back for success). See the ioctl(9e) manpage. - As pointed out in the example in the WDD, you can no longer dereference arg directly. You must use copyin() and copyout() to move the data around. Note that the buffer pointer argument to copyin() and copyout() must be cast to caddr_t. The cmds which were defined as _IOW will use copyin(), _IOR will use copyout(), and _IOWR will use both. - xx_strategy() (See the strategy(9e) manpage) - Note that xx_strategy should now return an int. The value is ignored, so return whatever you like. Zero is safe. - If you used to find your unit number from the buf struct this way [ unit_no = getminor(bp->b_dev); ] <-- uses b_dev field it must change to unit_no = getminor(bp->b_edev); <-- uses b_edev field - If you were using mb_mapalloc before, you should change that to using ddi_dma_buf_setup. If no "arg" is necessary, use (caddr_t) 0. You will need to use ddi_dma_htoc() to convert the dma handle into a dma cookie, from which you can get the kernel virtual dvma address to plug into your DMA engine address register. - If you are a DVMA device, and your dma engine has limited bits of resolution (such as the top 8 bits being registered rather than a counter), then you should set up a ddi_dma_lim_t structure describing these limits. Pass the address of this struct as the 5th argument to ddi_dma_buf_setup(). See ddi_dma_lim(9s). -You should define a new element of your unit struct to store the dma handle, since you will need it later to free the memory after the DMA is completed. - If you used to allocate your buffers uncached, you can now use ddi_dma_sync() whenever you must require cache consistency. You need not allocate your buffers uncached. - the semantics of timeout() and untimeout() have changed. timeout() returns an identifier, which must be saved to pass to untimeout(). This may require changes to your unit struct, if you stored pending timeout information in any other form than an int for each timeout there could be. - As stated above, save your buf address. - xx_poll() and xx_intr() - Use ddi_dma_free() where you used to use mb_mapfree(). - See the note above under xx_attach about eliminating xx_poll. - Make sure you are calling biodone with the new buffer address, not the old one, if you used to allocate your own buf struct. Step 4) Other General Notes **** Mutexes versus Interrupt Blocking **** Mutexes versus interrupts: Calls to spl() and splx() NO LONGER DO ANYTHING! All critical sections should be surrounded by mutex_enter()/ mutex_exit() pairs. Making a driver MT-safe: - In your cb_ops struct, change your driver compatibility flag (last struct element) thusly: D_NEW | D_MTSAFE, /* driver compatibility flag */ - define a mutex in your driver state (softc)structure something like this: mutex_t my_mutex; /* mutex for any op */ You can define multiple mutexes for different things you want to protect if you like. See cgthree.c for an example (it uses one mutex for the registers, another for the pixrect, and another for a busy flag). However, you can get by with just one. - In your attach routine, after the space for your mutex has been kmem_alloced, call mutex_init thusly: mutex_init(&my_units[unit_no]->my_mutex, "my mutex", MUTEX_DRIVER, (void *)0; - Wherever you want to protect something from access by anther thread, like a register or structure, protect that access by mutex_enter and mutex_exit calls. There are two points to note. NEVER nest mutex_enter/mutex_exit pairs. (That is, never call mutex_enter again after calling it once.) Also, it used to be safe to assume that no other thread could access anything in your interrupt routine, since you were spl'd to your interrupt level. Now, another thread could get in there. So you will need mutex_enter and mutex_exit calls around critical regions in your interrupt routines as well. Also, be very careful of items you used to initialize on the basis that no thread of control could be in your xx_read and xx_write routines at the same time. Now there could be such threads. - IF you need to know if a mutex is held, do this: ASSERT(MUTEX_HELD(&my_structure->my_mutex)); If the mutex is not held, the system will panic with an assertion failure.