GLDv3 Porting Guide

This guide describes how to port drivers to from GLDv2 architecture to GLDv3 architecture. For a complete guide to writing Solaris drivers, see the Writing Device Drivers book. For information on writing network drivers for the Solaris OS, see Chapter 19, "Drivers for Network Devices."

1 Overview

The explanations in this document assume that you have a GLDv2 driver that has already been written and tested, and you simply want to convert that GLDv2 driver to be a GLDv3 driver. No prior knowledge of GLDv2 or GLDv3 is assumed. The conversion process primarily involves declaring and initializing new data structures that are used when registering with the framework.

1.1 Advantages of GLDv3 Over GLDv2

The majority of code changes necessary to port GLDv2 drivers to GLDv3 simply involve translating the data
structures used and renaming some functions. GLDv3 provides several major advantages over GLDv2.

• Interrupt blanking
• Hardware checksum
• Link aggregation (802.3ad)
• VLANs (802.1q)

1.2 Porting Guidelines

This section provides some suggestions on how to port to GLDv3 most efficiently.

1.2.1 Overview of Essential Porting Steps

The following steps for porting from GLDv2 to GLDv3 must be done in order to get basic send and receive functionality to work with the driver.

1. Change the attach() entry point to use the mac_info_t structure instead of the gld_mac_info_t structure. Translate the structure fields.
2. Set up entry points in mac_t. Modify the functions that the entry points are set to so that the arguments are in the correct order.

   No mac_t

3. Modify the calls to unregister() to use the GLDv3 version.

   ethernet: VLANs are used - full frame size support is required

4. Update header files to include mac.h and mac_ether.h instead of gld.h.
5. Update the makefiles.

1.2.2 Overview of Additional Porting Steps

The following steps allow you to utilize the extended functionality in GLDv3. You should follow these steps as well as possible to allow for optimal driver performance under the GLDv3 framework.

1. Enable hardware blanking by setting up an additional entry point and providing a blanking function based on instructions from the NIC vendor.
2. Set up the checksum field of the mac_info_t data structure to enable hardware checksum.

1.2.3 Recommended Development Strategy

The most efficient strategy to follow for porting GLDv2 drivers to GLDv3 drivers includes the following steps:

1. Starting with the existing GLDv2 driver code, port the essential driver functions to GLDv3.
2. Make sure the updated driver compiles successfully using the mac.h header
   file and the misc/mac library.
3. Make sure the updated driver loads.
4. Check for link up.
5. Test receive and transmit.

   Add VLAN support (ethernet only)

6. Port the extended features of the GLDv3 drivers.
7. Test the extended features, such as interrupt blanking and hardware checksum,
   in the hardware test environment.
8. Perform stress and performance testing of the driver.

1.3 Code Examples

This guide shows code samples that you can reference. In addition, you should review drivers that have already been ported. GLDv3 drivers are available for viewing on the OpenSolaris project. A good driver to review is the bge driver. Take the following steps to view the bge driver:

1. On the OpenSolaris project site, click the Source Browser button to go to http://cvs.opensolaris.org/source/.
2. In the File Path text field, enter bge.
3. Make sure the onnv Project is selected.
4. Click the Search button.

1.4 List of Driver Functions to Modify

The following list shows the functions that must be modified during a port to GLDv3 for drivers that are structured similar to bge.

2 Driver Conversion Tasks

This section details the changes that must be made to the attach() and detach() entry points, as well as specifics of implementing other GLDv3 driver entry points.

2.1 attach() Changes

The attach() entry point attaches a device to the system.

2.1.1 Set Up State Information Data Structures

The global header file might have a data structure associated with the card for
per-card state information. If you find a declaration of a gld_mac_info_t pointer, change it to a mac_t pointer.

2.1.2 Set Up mac_info_t *

In GLDv2, the code uses a gld_mac_info_t pointer. In GLDv3, this needs to be removed and replaced with a mac_info_t pointer.

GLDv3 must allocation a mac_t structure, and then set the mac_info_t m_info pointer to mac_t *macp. The mac_info_t pointer can be used in the next step when setting up device specific functions.

bge_attach(dev_info_t *devinfo,
ddi_attach_cmd_t cmd)
gld_mac_info_t *macinfo; /* GLD */
bge_t *bgep; /* Our private data */
macinfo = gld_mac_alloc(devinfo);
ddi_set_driver_private(devinfo, macinfo);
bgep = kmem_zalloc(sizeof (*bgep), KM_SLEEP);
bgep->bge_guard = BGE_GUARD;
bgep->devinfo = devinfo;
bgep->macinfo = macinfo;
macinfo->gldm_private = (caddr_t)bgep;
...
macinfo->gldm_ident = bge_ident;
macinfo->gldm_type = DL_ETHER;
macinfo->gldm_minpkt = 0;
macinfo->gldm_maxpkt = ETHERMTU;
macinfo->gldm_addrlen = ETHERADDRL;
macinfo->gldm_saplen= -2;
macinfo->gldm_broadcast_addr=bge_broadcast_addr;
macinfo->gldm_vendor_addr=cidp->vendor_addr.addr;
macinfo->gldm_ppa= instance;
macinfo->gldm_devinfo= devinfo;
macinfo->gldm_cookie=(ddi_iblock_cookie_t)
    (uint64_t bgep->intr_pri;

2.1.3 Set Up mac_info_t Fields

Pointers to device specific functions that will be used by the generic layer must be initized. This is done by setting fields in the mac_info_t data structure, which is contained in the mac_t structure. This structure is later passed to mac_register().

The fields of the mac_info_t structure must be set. See also Section 2.5, "Converting MAC Structures."

2.1.4 Set Up Entry Points in mac_t

The fields of the mac_t structure need to be set to point to the corresponding entry points in the driver. See also Section 2.6, "Converting MAC Structure Entry Points."

2.1.5 Register With MAC Layer

At this point, the mac_t data structure is set up and can be registered. GLDv2 uses gld_register(). In GLDv3, this changes to mac_register().

2.2 detach() Changes

The detach() entry point detaches a device from the system.

2.2.1 Unregister

In GLDv2, get a pointer to the gld_mac_info_t structure from ddi_get_driver_private() and pass the pointer to gld_unregister().

In GLDv3, get a pointer to the bge_t structure from ddi_get_driver_private() and pass the pointer to mac_unregister().

2.3 _init() Changes

2.4 _fini() Changes

2.5 Converting MAC Structures

The GLDv3 mac_info_t structure is defined in sys/mac.h.

/*
 * Immutable information. (This may not be modified after registration).
 */
typedef struct mac_info_s {
	uint_t		mi_media;
	uint_t		mi_nativemedia;
	uint_t		mi_addr_length;
	uint8_t		*mi_unicst_addr;
	uint8_t		*mi_brdcst_addr;
} mac_info_t;

¹ bcopy(ether_brdcst, mip->mi_brdcst_addr, ETHERADDRL);

² bcopy(bgep>curr_addr.addr, mip->mi_unicst_addr, ETHERADDRL);

2.6 Converting MAC Structure Entry Points

2.7 Header Changes

In GLDv2, include gld.h. In GLDv3, include mac.h and mac_ether.h.

2.8 Makefile Changes

In GLDv2, link with misc/gld. In GLDv3, link with misc/mac.

2.9 Additional Required GLDv3 Functions

This section describes other GLDv3 functions and where these functions need to be called. These functions do not directly correspond to GLDv2 functions.

2.9.1 mac_resource_add()

extern mac_resource_handle_t
    mac_resource_add(mac_handle_t, mac_resource_t *);

The mac_resource_add() function should be called from the m_resources() entry ;point to register individual receive resources (commonly ring buffers of DMA descriptors) with the MAC module.

The returned mac_resource_handle_t value should then be suplied in calls to mac_rx().

The second argument to mac_resource_add() specifies the resource to be added. Resources are specified by the mac_resource_t structure. Currently, only resources of type MAC_RX_FIFO are supported. MAC_RX_FIFO resources are described by the mac_rx_fifo_t structure.

typedef struct mac_rx_fifo_s {
        mac_resource_type_t     mrf_type;       /* MAC_RX_FIFO */
        mac_blank_t             mrf_blank;
        void                    *mrf_arg;
        time_t                  mrf_normal_blank_time;
        uint_t                  mrf_normal_pkt_count;
} mac_rx_fifo_t;

The mrf_blank field of the mac_rx_fifo_t structure contains a mac_blank_t function. This function is meant to be used by the upper layers to control the interrupt rate of the device. The mrf_arg field is the device context that is meant to be used as the first argument to poll_blank().

The mrf_normal_blank_time field specifies the default interrupt interval, and the mrf_normal_pkt_count field specifies the packet count threshold. These parameters can be used as the second and third arguments to poll_blank when you need to revert to the default interrupt rate. You can also increase or decrease the interrupt rate by passing a multiple of these values as the last two arguments of poll_blank().

2.9.2 mac_tx_update()

extern void mac_tx_update(mac_handle_t)

The mac_tx_update() function is called by a driver to indicate that packet transmission resources are now available. The MAC module then sends a notification.

/*
 * Link state data (protected by genlock)
 * Defined in sys/mac.h
 */
typedef enum {
        LINK_STATE_UNKNOWN = -1,
        LINK_STATE_DOWN,
        LINK_STATE_UP
} link_state_t;

The attach() function should call mac_tx_update() to indicate that packet transmission resources are now available.

2.9.3 mac_rx()

extern void mac_rx(mac_handle_t, mac_resource_handle_t, mblk_t *);

Use the mac_rx() function to submit a chain of packets for reception. The chain of packets is contained in mblk_t structures. Fragments of the same packet should be linked together using the b_cont field. Separate packets should be linked using the b_next field of the leading fragment.

/*
 * Message block descriptor
 * Defined in sys/stream.h
 */
typedef struct	msgb {
	struct	msgb	*b_next;
	struct  msgb	*b_prev;
	struct	msgb	*b_cont;
	unsigned char	*b_rptr;
	unsigned char	*b_wptr;
	struct datab 	*b_datap;
	unsigned char	b_band;
	unsigned char	b_tag;
	unsigned short	b_flag;
	queue_t		*b_queue;	/* for sync queues */
} mblk_t;

If the packet chain was received by a registered resource, then the appropriate mac_resource_handle_t value should be supplied as the second argument to the function. The protocol stack uses this value as a hint when trying to load-spread across multiple CPUs. It is assumed that packets that belong to the same flow will always be received by the same resource.

If the resource is unknown or is unregistered, then NULL should be passed as the second argument.

2.9.4 mac_link_update()

extern void mac_link_update(mac_handle_t, link_state_t);

The mac_link_update() function should be called whenever the state of the media link changes. The second argument should be a value in the link_state_t enumeration shown above. The MAC module saves this value and sends a notification.

Calls to gld_linkstate() need to be replaced with calls to mac_link_update() to propagate MAC_NOTE_LINK notifications.

2.10 Enabling Additional GLDv3 Features

This section describes the procedure for enabling GLDv3 features. Note that zero copy and link aggregation (802.3ad) are supported without any additional code modifications.

2.10.1 Interrupt Blanking

In the m_resources() entry point, set up the mac_rx_fifo_t data structure to have mac_rx_fifo_t.mrf_blank set to point to xxchip_blank(), where _xx_chip_blank() makes the appropriate calls to ddi_put32().

2.10.2 Hardware Checksum

Set the mac_info_t.mi_cksum field to enable hardware checksum.

3 Adding Debug Statements

Do not use debug constructs such as the following:

#if_def xx_DEBUG
...debug code...
#endif

or

xxdebug_printf(...);

The above debug techniques are of limited value after initial development and are not consistent with coding standards. Be sure to use DTrace probes instead.

A great amount of documentation of DTrace exists, and already-instrumented code can be used for an example. use cscope or another tool to look around in the kernel code for calls to DTRACE_PROBE(), DTRACE_PROBE2(), and other similar calls.