2014/11/25

android 使用SMD(共享内存)作為跨processes溝通媒介

首先我也是參考大陸筆者網站, 最近念到SMD的部分! 一樣的Qualcomm的處理器裡面有許多subsystem 個別子處理器負責不同sub system的正常運行, 而我讀到的是負責Wifi/BT/FM的WCN36x0 chip! 很有趣的歷史背景, 因為WCN3680這個chip你可以看見是Qualcomm買下的 Atheros 所製作…………..

前言: 多核間的通訊

在智慧型手機SOC平台中,為了保證個別功能在平台上的流暢運行,都會依照功能使用專門的處理器來處理。如在Qucalcomm MSM 7K 平台上,就包含了4 個處理器內核,Qucalcomm MSM 7K 平台採用ARM 9(mARM,modemARM)處理基帶業務;採用mDSP(Modem DSP)來處理協議protocol的事情;採用ARM 11(aARM ,application ARM)來負責Linux 操作系統的運行;採用aDSP(Application DSP)來處理多媒體業務方面的編/解碼加速工作。

在2010 年末,smartphone or Tablet 終端的CPU 得到了快速發展,高端的smart devices終端已經採用了Cortex-A8 的雙核處理器。而在平板電腦上,Nvidia 開發的基於Cortex-A9 的雙核處理器Tegra 2 則成了市場的寵兒。通信離不開memory記憶體的操作,在Qualcomm平台上,記憶體一般分為3種:基帶內存(Modem Memory)、應用記憶體(Application Memory)和共享記憶體,其中系統MPU保護基帶記憶體不被aARM接入,ARM MMU保護應用記憶體不被mARM接入。

共享內存share-meory

在Linux 中,實現進程通信的機制有很多種,如信號signal、管道Pipeline、信號量Semaphore 、消息隊列Message queue、共享內存share-meory和套接字socket等,但共享內存的方式效率最高。

共享內存是多核通信的物理基礎,其實現主要包括3 個部分:共享內存驅動(SMD,Shared Memory Driver)、共享內存狀態機(SMSM,Shared Memory State Machine)和共享內存管理器(SMEM,Shared Memory Manager)。
其中
SMD 用於多核之間的數據通信;
SMSM用於多核之間的狀態通信;
SMEM是一個底層的協議,是物理RAM共享內存的管理接口,是SMD和SMSM的基礎。

SMEM 具有兩種分配模式:動態SMEM 和靜態SMEM,
動態SMEM 根據需要實時分配,靜態SMEM則會預先分配。

SMEM的主要接口為:smem_alloc()、smem_find()、smem_init()等。
SMEM、SMD、SMSM的實現都需要硬件平台廠商提供支持。

同步與互斥- Synchronization and mutual exclusion

Share meomry用到了自旋鎖(spinlock)和互斥鎖(Mutual exclusion)的概念。
自旋鎖(spinlock)是Linux 內核的同步機制之一,與互斥鎖(Mutual exclusion)類似,但自旋鎖使用者一般保持鎖的時間非常短。自旋鎖的效率遠高於互斥鎖。
自旋鎖的定義位於 xxx\include\linux\spinlock_*.h文件中。

typedef struct {volatile unsigned int lock; // unsigned integer} Raw_spinlock_t;

lock雖然被定義為無符號整數,但是實際上被當做有符號整數使用。
slock值為1 代表鎖未被佔用,值為0 或負數代表鎖被佔用。初始化時slock被置為1。
與信號量Semaphore和讀寫信號量導致調用者睡眠不同,自旋鎖spinlock不會引起調用者睡眠。如果自旋鎖已被別的執行單元保持,調用者就一直循環以確定是否該自旋鎖的保持者已經釋放了鎖。

由於自旋鎖spinlock適用的訪問共享資源的時間非常短, 導致自旋鎖spinlock通常應用於中斷上下文Interrupt Context訪問和對共享資源訪問文件非常短的場景中,如果被保護的共享資源在進程process上下文訪問,則應使用信號量。

與信號量Semaphore和讀寫信號量rw_semaphore在保持期間可以被搶占的情況不同,自旋鎖spinlock在保持期間是搶占失效的,自旋鎖只有在內核可搶占或SMP(Symmetrical Multi-Processing)的情況下才真正需要,在單CPU且不可搶占的內核下,自旋鎖spinlock的所有操作都是空操作。

由於smart devices平台上通常存在多個CPU 或DSP,自旋鎖的運用就顯得非常重要。
在mach-msm Kernel裡的SMD 和SMSM的實現上,自旋鎖spinlock主要運用於中斷處理、信道列表List_of_WLAN_channels和信道狀態Channel State Information的變更過程中,自旋鎖的定義如下:

/* the spinlock is used to synchronize between the * irq handler and code that mutates the channel * list or fiddles with channel state */static DEFINE_SPINLOCK(smd_lock);DEFINE_SPINLOCK(smem_lock);

互斥鎖主要用於實現Linux 內核中的互斥訪問功能,在mach-msm Kernel的SMD 的實現上,互斥鎖主要用於SMD 信道的打開或關閉過程。定義如下:

/* the mutex is used during open() and close() * operations to avoid races while creating or * destroying smd_channel structures */static DEFINE_MUTEX(smd_creation_mutex);

SMD 控制訊號傳遞

在Linux 中,基於SMD 的訊號傳遞是以channel的形式作為一個設備存在的,作為一種雙向信道Channel,其接口的實現遵循Linux 設備驅動規範。
在Qucalcomm 平台上,SMD 的緩衝buffer大小為8192bit,最大信道數為64,SMD的頭head大小為20bit。

SMD 的相關代碼實現主要位於 arch\arm\mach-msm目錄下。
主要文件包括: smd.c、smd_nmea.c、smd_qmi.c、smd_rpcrouter.c、smd_rpcrouter_clients.c、smd_rpcrouter_device.c、smd_rpcrouter_servers.c、smd_tty.c等。

SMD 信道Channel需要同時維護接收信道、發送信道的狀態和數據信息,SMD 的信道定義如下:

smd_private.h

struct smd_channel {	volatile void __iomem *send; /* some variant of smd_half_channel 發送握手channel訊息 */	volatile void __iomem *recv; /* some variant of smd_half_channel 接收握手channel訊息*/	unsigned char *send_data; //發送chaneel 數據資料	unsigned char *recv_data; //接收chaneel 數據資料	unsigned fifo_size;	unsigned fifo_mask;	struct list_head ch_list;	unsigned current_packet;	unsigned n;	void *priv;	void (*notify)(void *priv, unsigned flags);	int (*read)(smd_channel_t *ch, void *data, int len, int user_buf); //讀取	int (*write)(smd_channel_t *ch, const void *data, int len,  //寫入			int user_buf);	int (*read_avail)(smd_channel_t *ch); //可否讀取	int (*write_avail)(smd_channel_t *ch); //可否寫入	int (*read_from_cb)(smd_channel_t *ch, void *data, int len,			int user_buf);	void (*update_state)(smd_channel_t *ch);	unsigned last_state;	void (*notify_other_cpu)(smd_channel_t *ch);	char name[20];	struct platform_device pdev;	unsigned type;	int pending_pkt_sz;	char is_pkt_ch;	/*	 * private internal functions to access *send and *recv.	 * never to be exported outside of smd	 */	struct smd_half_channel_access *half_ch;};

SMD 初始化過程

msm_smd_init()

	for (i = 0; i < NUM_SMD_SUBSYSTEMS; ++i) {		remote_info[i].remote_pid = i;		remote_info[i].free_space = UINT_MAX;		INIT_WORK(&remote_info[i].probe_work, smd_channel_probe_worker);		INIT_LIST_HEAD(&remote_info[i].ch_list);	}

smd_channel_probe_worker()
– Scan for newly created SMD channels and init local structures so the channels are visable to local clients
==> scan_alloc_table(shared, &(r_info->ch_allocated[SMEM_NUM_SMD_STREAM_CHANNELS]), SEC_ALLOC_TBL, tbl_size / sizeof(*shared), r_info);

scan_alloc_table()

/** * scan_alloc_table - Scans a specified SMD channel allocation table in SMEM for newly created channels that need to be made locally visable * @shared: pointer to the table array in SMEM * @smd_ch_allocated: pointer to an array indicating already allocated channels * @table_id: identifier for this channel allocation table * @num_entries: number of entries in this allocation table * @r_info: pointer to the info structure of the remote proc we care about * The smd_probe_lock must be locked by the calling function.  Shared and smd_ch_allocated are assumed to be valid pointers. */static void scan_alloc_table(struct smd_alloc_elm *shared, char *smd_ch_allocated, int table_id, unsigned num_entries, struct remote_proc_info *r_info){	unsigned n;	uint32_t type;	for (n = 0; n < num_entries; n++) {		if (smd_ch_allocated[n])			continue;		/*		 * channel should be allocated only if APPS processor is involved		 */		type = SMD_CHANNEL_TYPE(shared[n].type);		if (!pid_is_on_edge(type, SMD_APPS) ||				!pid_is_on_edge(type, r_info->remote_pid))			continue;		if (!shared[n].ref_count)			continue;		if (!shared[n].name[0])			continue;		if (!smd_initialized && !smd_edge_inited(type)) {			SMD_INFO(				"Probe skipping proc %d, tbl %d, ch %d, edge not inited r_info->remote_pid, table_id, n);			continue;		}		if (!smd_alloc_channel(&shared[n], table_id, r_info))			smd_ch_allocated[n] = 1;		else			SMD_INFO(				"Probe skipping proc %d, tbl %d, ch %d, not allocated r_info->remote_pid, table_id, n);	}}

smd_alloc_channel(): 分配channel

/** * smd_alloc_channel() - Create and init local structures for a newly allocated SMD channel * @alloc_elm: the allocation element stored in SMEM for this channel * @table_id: the id of the table this channel resides in. 1 = first table, 2 = *		seconds table, etc * @r_info: pointer to the info structure of the remote proc for this channel * @returns: -1 for failure; 0 for success */static int smd_alloc_channel(struct smd_alloc_elm *alloc_elm, int table_id,				struct remote_proc_info *r_info){	struct smd_channel *ch;	ch = kzalloc(sizeof(struct smd_channel), GFP_KERNEL);	if (ch == 0) {		pr_err("smd_alloc_channel() out of memory\n");		return -1;	}	ch->n = alloc_elm->cid;	ch->type = SMD_CHANNEL_TYPE(alloc_elm->type);	if (smd_alloc_v2(ch, table_id, r_info) && smd_alloc_v1(ch)) {		kfree(ch);		return -1;	}	ch->fifo_mask = ch->fifo_size - 1;	/* probe_worker guarentees ch->type will be a valid type */	if (ch->type == SMD_APPS_MODEM)		ch->notify_other_cpu = notify_modem_smd;	else if (ch->type == SMD_APPS_QDSP)		ch->notify_other_cpu = notify_dsp_smd;	else if (ch->type == SMD_APPS_DSPS)		ch->notify_other_cpu = notify_dsps_smd;	else if (ch->type == SMD_APPS_WCNSS)		ch->notify_other_cpu = notify_wcnss_smd;	else if (ch->type == SMD_APPS_RPM)		ch->notify_other_cpu = notify_rpm_smd;	if (smd_is_packet(alloc_elm)) {   //如果是 packet包 channel		ch->read = smd_packet_read;		ch->write = smd_packet_write;		ch->read_avail = smd_packet_read_avail;		ch->write_avail = smd_packet_write_avail;		ch->update_state = update_packet_state;		ch->read_from_cb = smd_packet_read_from_cb;		ch->is_pkt_ch = 1;	} else {                     //如果是 stream流 channel		ch->read = smd_stream_read;		ch->write = smd_stream_write;		ch->read_avail = smd_stream_read_avail;		ch->write_avail = smd_stream_write_avail;		ch->update_state = update_stream_state;		ch->read_from_cb = smd_stream_read;	}	memcpy(ch->name, alloc_elm->name, SMD_MAX_CH_NAME_LEN);	ch->name[SMD_MAX_CH_NAME_LEN-1] = 0;	ch->pdev.name = ch->name;	ch->pdev.id = ch->type;	SMD_INFO("smd_alloc_channel() '%s' cid=%d\n",		 ch->name, ch->n);	mutex_lock(&smd_creation_mutex);/互斥锁  	list_add(&ch->ch_list, &smd_ch_closed_list); //将信道添加到“smd_ch_closed_list”列表中	mutex_unlock(&smd_creation_mutex);	platform_device_register(&ch->pdev); //注册设备	if (!strncmp(ch->name, "LOOPBACK", 8) && ch->type == SMD_APPS_MODEM) {		/* create a platform driver to be used by smd_tty driver		 * so that it can access the loopback port		 */		loopback_tty_pdev.id = ch->type;		platform_device_register(&loopback_tty_pdev);	}	return 0;}

開啟channel- Open the SMD channel

常見在wifi driver初始化過程中會呼叫, 或是該driver需要SMD的話就會打開SMD啦…
為了打開一個信道channel,首先要判斷SMD 信道是否已經初始化。
如果SMD 信道已經初始化,就根據信道channel名獲得信道channel,將信道channel加入到“smd_ch_list”信道列表中並設置該信道的狀態為SMD_SS_OPENING,然後調用notify_other_smd()函數通知其他的信道該信道已經激活enable。

在默認default情況下,其信道類型為SMD_APPS_MODEM,打開一個SMD信道的實現如下:

int smd_named_open_on_edge(const char *name, uint32_t edge,			   smd_channel_t **_ch,			   void *priv, void (*notify)(void *, unsigned)){	struct smd_channel *ch;	unsigned long flags;	if (smd_initialized == 0 && !smd_edge_inited(edge)) {  //判断SMD 信道是否已初始化 		SMD_INFO("smd_open() before smd_init()\n");		return -ENODEV;	}	SMD_DBG("smd_open('%s', %p, %p)\n", name, priv, notify);	ch = smd_get_channel(name, edge);  //獲得channel頻道	if (!ch) {		/* check closing list for port */		spin_lock_irqsave(&smd_lock, flags); //自旋锁  		list_for_each_entry(ch, &smd_ch_closing_list, ch_list) {			if (!strncmp(name, ch->name, 20) &&				(edge == ch->type)) {				/* channel exists, but is being closed */				spin_unlock_irqrestore(&smd_lock, flags);				return -EAGAIN;			}		}		/* check closing workqueue list for port */		list_for_each_entry(ch, &smd_ch_to_close_list, ch_list) {			if (!strncmp(name, ch->name, 20) &&				(edge == ch->type)) {				/* channel exists, but is being closed */				spin_unlock_irqrestore(&smd_lock, flags);				return -EAGAIN;			}		}		spin_unlock_irqrestore(&smd_lock, flags);		/* one final check to handle closing->closed race condition */		ch = smd_get_channel(name, edge);		if (!ch)			return -ENODEV;	}	if (notify == 0)		notify = do_nothing_notify;	ch->notify = notify;	ch->current_packet = 0;	ch->last_state = SMD_SS_CLOSED;	ch->priv = priv;	if (edge == SMD_LOOPBACK_TYPE) {		ch->last_state = SMD_SS_OPENED;		ch->half_ch->set_state(ch->send, SMD_SS_OPENED);		ch->half_ch->set_fDSR(ch->send, 1);		ch->half_ch->set_fCTS(ch->send, 1);		ch->half_ch->set_fCD(ch->send, 1);	}	*_ch = ch;	SMD_DBG("smd_open: opening '%s'\n", ch->name);	spin_lock_irqsave(&smd_lock, flags); //自旋锁 	if (unlikely(ch->type == SMD_LOOPBACK_TYPE))		list_add(&ch->ch_list, &smd_ch_list_loopback); //将信道添加到“smd_ch_list_loopback”列表中	else		list_add(&ch->ch_list, &remote_info[edge_to_pids[ch->type].remote_pid].ch_list);	SMD_DBG("%s: opening ch %d\n", __func__, ch->n);	if (edge != SMD_LOOPBACK_TYPE)		smd_state_change(ch, ch->last_state, SMD_SS_OPENING); //信道状态变更 	spin_unlock_irqrestore(&smd_lock, flags);	return 0;}EXPORT_SYMBOL(smd_named_open_on_edge);

關閉信道- smd_close()

關閉信道的操作相對簡單,首先將信道從“smd_ch_list”信道列表中刪除,然後將信道狀態設置為SMD_SS_CLOSED,並將信道添加到“smd_ch_closed_list”信道列表中即可。關閉SMD 信道的實現如下:

int smd_close(smd_channel_t *ch){	unsigned long flags;	if (ch == 0)		return -1;	SMD_INFO("smd_close(%s)\n", ch->name);	spin_lock_irqsave(&smd_lock, flags); //自旋锁	list_del(&ch->ch_list); //从打开信道列表中去除该信道	if (ch->n == SMD_LOOPBACK_CID) {		ch->half_ch->set_fDSR(ch->send, 0);		ch->half_ch->set_fCTS(ch->send, 0);		ch->half_ch->set_fCD(ch->send, 0);		ch->half_ch->set_state(ch->send, SMD_SS_CLOSED);	} else		ch_set_state(ch, SMD_SS_CLOSED);   //设置信道状态	if (ch->half_ch->get_state(ch->recv) == SMD_SS_OPENED) {		list_add(&ch->ch_list, &smd_ch_closing_list);		spin_unlock_irqrestore(&smd_lock, flags);	} else {		spin_unlock_irqrestore(&smd_lock, flags);		ch->notify = do_nothing_notify;		mutex_lock(&smd_creation_mutex);  //互斥锁		list_add(&ch->ch_list, &smd_ch_closed_list); //将信道添加至关闭信道列表 		mutex_unlock(&smd_creation_mutex);	}	return 0;}EXPORT_SYMBOL(smd_close);

channel內容讀取-1. smd_packet_read

packet channel的內容讀取涉及緩衝的複制、與其他SMD 的消息通信和包狀態的更新。
從packet channel讀取數據的實現如下:

static int smd_packet_read(smd_channel_t *ch, void *data, int len, int user_buf){	unsigned long flags;	int r;	if (len < 0)		return -EINVAL;	if (len > ch->current_packet)		len = ch->current_packet;	r = ch_read(ch, data, len, user_buf); //读取数据	if (r > 0)		if (!read_intr_blocked(ch))			ch->notify_other_cpu(ch);	spin_lock_irqsave(&smd_lock, flags); //自旋锁	ch->current_packet -= r;	update_packet_state(ch); //更新包状态 	spin_unlock_irqrestore(&smd_lock, flags);	return r;

channel內容讀取-2. smd_stream_read

流信道的內容讀取非常簡單,只需要調用ch_read()函數讀取數據並通知其他SMD 該信道處於打開狀態即可。
smd_stream_read 流信道讀取的實現如下:

static int smd_stream_read(smd_channel_t *ch, void *data, int len, int user_buf){	int r;	if (len < 0)		return -EINVAL;	r = ch_read(ch, data, len, user_buf); //读取数据 	if (r > 0)		if (!read_intr_blocked(ch))			ch->notify_other_cpu(ch); //通知其他CPU,该信道处于激活状态。	return r;}

流信道和包信道在讀取信道數據時,
都需要調用ch_read()函數來實現真正的數據讀取

ch_read()函數的實現如下:

/* basic read interface to ch_read_{buffer,done} used * by smd_*_read() and update_packet_state() * will read-and-discard if the _data pointer is null */static int ch_read(struct smd_channel *ch, void *_data, int len, int user_buf){	void *ptr;	unsigned n;	unsigned char *data = _data;	int orig_len = len;	int r = 0;	while (len > 0) {		n = ch_read_buffer(ch, &ptr); //將channel的		if (n == 0)			break;		if (n > len)			n = len;		if (_data) {			if (user_buf) { //如果是來自user space的, 但我們目前看到的是 kernel 內部呼叫所以為else..				r = copy_to_user(data, ptr, n);				if (r > 0) {					pr_err("%s: "						"copy_to_user could not copy "						"%i bytes.\n",						__func__,						r);				}			} else				memcpy(data, ptr, n); //数据复制		}		data += n;		len -= n;		ch_read_done(ch, n); //读取完成	}	return orig_len - len;}

信道寫入-smd_packet_write

在信道的數據寫入方面,同樣分為流信道數據的寫入和包信道數據的寫入兩種類型。向包信道中寫入數據的過程和讀取數據不同,在寫入數據前,要首先利用smd_stream_write_avail()函數判斷是否有數據可供寫入,在確定有可供寫入的數據的情況下才調用sm​​d_stream_write()函數執行數據的寫入操作。包信道寫入數據的實現如下

static int smd_packet_write(smd_channel_t *ch, const void *_data, int len,				int user_buf){	int ret;	unsigned hdr[5];	SMD_DBG("smd_packet_write() %d -> ch%d\n", len, ch->n);	if (len < 0)		return -EINVAL;	else if (len == 0)		return 0;	if (smd_stream_write_avail(ch) < (len + SMD_HEADER_SIZE))		return -ENOMEM;	hdr[0] = len;	hdr[1] = hdr[2] = hdr[3] = hdr[4] = 0;	ret = smd_stream_write(ch, hdr, sizeof(hdr), 0); //流写入	if (ret < 0 || ret != sizeof(hdr)) {		SMD_DBG("%s failed to write pkt header: "			"%d returned\n", __func__, ret);		return -1;	}	ret = smd_stream_write(ch, _data, len, user_buf);	if (ret < 0 || ret != len) {		SMD_DBG("%s failed to write pkt data: "			"%d returned\n", __func__, ret);		return ret;	}	return len;}

smd_stream_write流信道數據的寫入
和包信道數據的寫入不同,
其首先獲得下一段可用緩衝的指針,然後執行內存複製,流信道寫入數據的實現如下:

static int smd_stream_write(smd_channel_t *ch, const void *_data, int len,				int user_buf){	void *ptr;	const unsigned char *buf = _data;	unsigned xfer;	int orig_len = len;	int r = 0;	SMD_DBG("smd_stream_write() %d -> ch%d\n", len, ch->n);	if (len < 0)		return -EINVAL;	else if (len == 0)		return 0;	while ((xfer = ch_write_buffer(ch, &ptr)) != 0) { //写入数据		if (!ch_is_open(ch)) {			len = orig_len;			break;		}		if (xfer > len)			xfer = len;		if (user_buf) {			r = copy_from_user(ptr, buf, xfer);			if (r > 0) {				pr_err("%s: "					"copy_from_user could not copy %i "					"bytes.\n",					__func__,					r);			}		} else			memcpy(ptr, buf, xfer);		ch_write_done(ch, xfer); //完成写入数据		len -= xfer;		buf += xfer;		if (len == 0)			break;	}	if (orig_len - len)		ch->notify_other_cpu(ch);	return orig_len - len;}

更仔細看 ch_write_buffer()

會提供指標給呼叫者, 也回傳寫入數目數字

static unsigned ch_write_buffer(struct smd_channel *ch, void **ptr){	unsigned head = ch->half_ch->get_head(ch->send); //得到head 頭的數字	unsigned tail = ch->half_ch->get_tail(ch->send); //得到tail 尾的數字	*ptr = (void *) (ch->send_data + head);   //將channel的傳送資料位置加上頭的數字, 則此指標就是可以寫入資料的位置	if (head < tail) {		return tail - head - SMD_FIFO_FULL_RESERVE; //如果頭數字反而比較小, 表示只有尾到頭這段是可以寫入(還要再扣掉保留區)	} else {		if (tail < SMD_FIFO_FULL_RESERVE)			return ch->fifo_size + tail - head - SMD_FIFO_FULL_RESERVE; //如果頭比尾的數字大, 可寫入的位置就是全部-頭+尾(還要再扣掉保留區)		else			return ch->fifo_size - head;	}}

FIFO 示意圖

更仔細看 ch_read_buffer()

會在FIFO中, 提供一個指標和可讀數據長度

static unsigned ch_read_buffer(struct smd_channel *ch, void **ptr){	unsigned head = ch->half_ch->get_head(ch->recv); //得到head 頭的數字	unsigned tail = ch->half_ch->get_tail(ch->recv); //得到tail 尾的數字	*ptr = (void *) (ch->recv_data + tail); //將channel的接受資料位置加上尾的數字, 則此指標就是可以接收資料的位置	if (tail <= head)		return head - tail; //由於要接收, 所以直觀上若尾比頭小, 表示尾到頭這段是可以接收的	else		return ch->fifo_size - tail; //反之, 由於要接收, 所以若頭比尾小, 表示全部扣掉尾那段是可以接收的}

SMD 是多核通信的基礎,
通過SMD,可以為系統提供RPC、DIAG、AT命令、NMEA(GPS數據)、數據服務、撥號等服務。

沒有留言:

張貼留言