Dust8 的博客

读书百遍其义自见

0%

起因

做人脸动作活体检测的时候使用手机拍摄视频, 视频用opencv打开出现了上下颠倒的情况.
当前opencv版本是4.6.0.66, 使用版本4.5.5.64版本正常.

分析

使用ffmpeg查看信息

1
ffprobe -print_format json -select_streams v -show_streams -i video_20221124_135508.mp4

-print_format json 是输出的格式为json. -select_streams v是选择视频流,v是视频, -show_streams 是显示流信息. -i video_20221124_135508.mp4是输入文件地址.
输出结果:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
"streams": [
{
"index": 0,
"width": 1920,
"height": 1080,
"coded_width": 1920,
"coded_height": 1080,
"side_data_list": [
{
"side_data_type": "Display Matrix",
"rotation": 90
}
]
}
]
}

rotation是一个十进制数, 指定视频在显示之前应逆时针旋转的度数.
这个解释来源于ffmpeg-display_rotation参数.
还有一个叫rotate也是度数, 官方解释还未找到, 它们的关系在opencv源代码里面可以看出.
rotate等于负的rotation, 如果rotate小于0, 则rotate等于rotate加360度.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
 // https://github.com/opencv/opencv/blob/da4ac6b7eff2e8869567e4faaff73312f9e1ef57/modules/videoio/src/cap_ffmpeg_impl.hpp#L1777

void CvCapture_FFMPEG::get_rotation_angle()
{
rotation_angle = 0;
#if LIBAVFORMAT_BUILD >= CALC_FFMPEG_VERSION(57, 68, 100)
const uint8_t *data = 0;
data = av_stream_get_side_data(video_st, AV_PKT_DATA_DISPLAYMATRIX, NULL);
if (data)
{
rotation_angle = -cvRound(av_display_rotation_get((const int32_t*)data));
if (rotation_angle < 0)
rotation_angle += 360;
}
#elif LIBAVUTIL_BUILD >= CALC_FFMPEG_VERSION(52, 94, 100)
AVDictionaryEntry *rotate_tag = av_dict_get(video_st->metadata, "rotate", NULL, 0);
if (rotate_tag != NULL)
rotation_angle = atoi(rotate_tag->value);
#endif
}

opencv自带的旋转

搜索opencvIssues可以知道, 大概是4.5版才加入的默认自动旋转视频, 使用的是ffmpeg的后端.

1
2
3
4
5
# 默认是自动旋转, 可以设置为0来取消
cap.set(cv2.CAP_PROP_ORIENTATION_AUTO, 0)

# 通过CAP_PROP_ORIENTATION_META可以获取到rotate
orientation = int(cap.get(cv2.CAP_PROP_ORIENTATION_META))

自动旋转的源代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
// https://github.com/opencv/opencv/blob/97c6ec6d49cb78321eafe6fa220ff80ebdc5e2f4/modules/videoio/src/cap_ffmpeg.cpp#L144
void rotateFrame(cv::Mat &mat) const
{
bool rotation_auto = 0 != getProperty(CAP_PROP_ORIENTATION_AUTO);
int rotation_angle = static_cast<int>(getProperty(CAP_PROP_ORIENTATION_META));

if(!rotation_auto || rotation_angle%360 == 0)
{
return;
}

cv::RotateFlags flag;
if(rotation_angle == 90 || rotation_angle == -270) { // Rotate clockwise 90 degrees
flag = cv::ROTATE_90_CLOCKWISE;
} else if(rotation_angle == 270 || rotation_angle == -90) { // Rotate clockwise 270 degrees
flag = cv::ROTATE_90_COUNTERCLOCKWISE;
} else if(rotation_angle == 180 || rotation_angle == -180) { // Rotate clockwise 180 degrees
flag = cv::ROTATE_180;
} else { // Unsupported rotation
return;
}

cv::rotate(mat, mat, flag);
}

自定义旋转

如果是其他的opencv不支持旋转的后端, 可以自己实现.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# 方式1
cv2.rotate()

# 方式2
cv2.flip()

# 方式3
def rotate_bound(image, angle):
# grab the dimensions of the image and then determine the
# center
(h, w) = image.shape[:2]
(cX, cY) = (w // 2, h // 2)
# grab the rotation matrix (applying the negative of the
# angle to rotate clockwise), then grab the sine and cosine
# (i.e., the rotation components of the matrix)
# getRotationMatrix2D, angle正值表示逆时针旋转(坐标原点假定为左上角)
M = cv2.getRotationMatrix2D((cX, cY), -angle, 1.0)
cos = np.abs(M[0, 0])
sin = np.abs(M[0, 1])
# compute the new bounding dimensions of the image
nW = int((h * sin) + (w * cos))
nH = int((h * cos) + (w * sin))
# adjust the rotation matrix to take into account translation
M[0, 2] += (nW / 2) - cX
M[1, 2] += (nH / 2) - cY
# perform the actual rotation and return the image
return cv2.warpAffine(image, M, (nW, nH))

参考链接

起因

同时在vue3中播放多个视频, 还能翻页.

问题

问题1.多个播放器

解决: 按照videojs官方文档的vue集成教程熟悉下, 在封装成组件, 一个组件有一个播放器, 循环就支持多个. 通过ref来自动生成id.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
<template>
<div>
<a href="#" class="group" v-for="item in items">
<div class="aspect-w-1 aspect-h-1 w-full overflow-hidden">
<video-player :options="item.videoOptions" />
</div>
<h3 class="mt-1 text-lg font-medium text-gray-900">{{ item.name }}</h3>
</a>
</div>
</template>

<script setup>
import { ref } from "vue";
import VideoPlayer from '@/components/video-player.vue'

const items = [
{
name:"湖南卫视",
videoOptions: {
autoplay: true,
muted: true,
controls: true,
sources: [
{
src: 'http://219.151.31.38/liveplay-kk.rtxapp.com/live/program/live/hnwshd/4000000/mnf.m3u8',
type: 'application/x-mpegURL'
}
]
}
}]
</script>

问题2.翻页播放

翻页后videojs生成的id未变, 视频还是更改前的. vue组件传进去的参数已经发生改变, 但videojs实例player的播放地址没有改变. 看官方文档有个player.src()方法可以改变播放地址. 却出现一直请求.m3u8文件, 未请求.ts视频文件.

解决:
1.在组件内通过watch来监听参数变化, 有变动通过player.src()来设置新的播放地址.
2.由于vue3 默认使用代理来声明变量, playersrc()就出现了问题, 可以把player声明为普通变量, 可以参考video.js/issues/7418.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
// video-player.vue
<script>
import videojs from 'video.js';
import "video.js/dist/video-js.css";
import zh from "video.js/dist/lang/zh-CN.json";
import { onBeforeUnmount, onMounted, watch, ref } from 'vue';

videojs.addLanguage('zh-CN', zh);

export default {
name: 'LjVideoPlayer',
props: {
options: {
type: Object,
default() {
return {};
}
}
},
setup(props) {
let videoPlayer = ref(null)
let player =null;

const onChange = () => {
if (player) {
player.src(props.options.sources)
}
}

const dispose = () => {
if (player) {
player.dispose()
}
}
onMounted(() => {
player = videojs(videoPlayer.value, props.options, () => {
// player.log('onPlayerReady', this);
});
})

onBeforeUnmount(() => {
if (player) {
player.dispose();
}
})

watch(props, () => {
onChange()
})

return { dispose, videoPlayer,}
},
}
</script>

<template>
<div>
<video ref="videoPlayer" class="video-js vjs-fluid vjs-fill" webkit-playsinline="true" playsinline="true"></video>
</div>
</template>

<style scoped>
video {
height: 100%;
width: 100%;
min-width: 100px;
min-height: 100px;
display: block;
}
</style>

问题3. vue3没有$refs

videojs初始化需要元素, 通过ref来指定后, 在取值时需要声明和导出.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<script>
export default {
setup(){
// 声明
let videoPlayer = ref(null)
let player =null

onMounted(() => {
player = videojs(videoPlayer.value, props.options, () => {
// player.log('onPlayerReady', this);
});
})
// 导出
return {videoPlayer,}
}
}
</script>

<template>
<div>
<video ref="videoPlayer"></video>
</div>
</template>

问题4. 中文语言

官方文档只说了设置语言, 却没说如何引入中文语言包. 没有说明vue如何引入中文包, 在https://github.com/videojs/video.js/issues/7986可以知道.

1
2
// 引入中文
import zh from "video.js/dist/lang/zh-CN.json";

1
2
3
4
5
6
7
8
9
10
11
12
13
// 初始化的时候设置中文
{
autoplay: true,
muted: true,
controls: true,
language: 'zh-CN',
sources: [
{
src: src,
type: 'application/x-mpegURL'
}
]
}

问题5. 重复使用一个弹出框来播放不同视频

一种是弹出生成播放实例, 关闭后销毁. 一种是重新设置视频地址.
下面说说不销毁的, 如果不销毁, 那么需要使用暂停, 会一直请求视频地址(m3u8)来缓存. 可以设置视频地址为空字符串, 来禁止一直请求.

1
2
3
player.pause()
player.src('')
player.load()

参考链接

起因

前端有个树形结构需要显示, 层级比较多, 达到7层, 还是需要自适应大小. 现有的组件要不层级不够, 例如像省市县这样的级联, 只有3层, 如果做成7层, 在移动端显示效果不好.
在网上发现yangjingyu/vs-tree比较符合要求, 上面用面包屑显示层级, 下面只显示最后一层的列表. 不过在vue3上有些问题. 刚好最近接触了hooks, 还了解了Headless UI. 把逻辑和ui进行分离, 以前看过(用积木理论设计一个灵活好用的Carousel走马灯组件), 也想过, 只是不知道这样一个统一的名称.
目前常见的组件通过slot来定制化还是不够灵活.
灵活性:

1
手写逻辑和ui > hooks + 手写ui > ui框架组件

当然灵活性越大, 工作量也越大.

实例

输入: data 是一个扁平的树, id为”_”的是根节点.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
"_": {
"id": "_",
"name": "_",
"children": ["1", "2"]
},
"1": {
"id": "1",
"name": "1",
"children": []
},
"2": {
"id": "2",
"name": "2",
"children": []
}
}

输出: 导出4个变量, breadcrumbNodes,breadcrumbNodeClick,是上面面包屑的节点列表, 和点击函数, currentNodes, currentNodeClick 是下面的节点列表和点击函数.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# useTree.js
import { ref } from "vue";

const useTree = (data) => {
const nodesMap = ref(data);
const breadcrumbNodes = ref([]);
const currentNodes = ref([]);

const updateCurrentNodes = (node) => {
const nodes = [];
nodesMap.value[node.id].children.forEach((element) => {
nodes.push(nodesMap.value[element]);
});
currentNodes.value = nodes;
};
const breadcrumbNodeClick = (index) => {
// 更新头
breadcrumbNodes.value = breadcrumbNodes.value.slice(0, index + 1);

// 更新列表
const node = breadcrumbNodes.value[index];
updateCurrentNodes(node);
};

const currentNodeClick = (index) => {
const node = currentNodes.value[index];
if (!node.children || !node.children.length) {
console.log("no children");
return;
}

// 更新头
breadcrumbNodes.value.push(node);

// 更新列表
updateCurrentNodes(node);
};

breadcrumbNodes.value.push(nodesMap.value["_"]);
updateCurrentNodes(nodesMap.value["_"]);

return {
breadcrumbNodes,
currentNodes,
breadcrumbNodeClick,
currentNodeClick,
};
};
export default useTree;

使用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
<script setup>
import useTree from './useTree'

const data2 = {}
const { breadcrumbNodes, currentNodes, breadcrumbNodeClick, currentNodeClick } = useTree(data2)

const listclick2 = (index) => {
const node = currentNodes.value[index]
console.log('listclick2', index, node.name)
currentNodeClick(index)
}

</script>
<template>
<div>
<div class="tree-box">
<div class="tree-breadcrumb">
<span v-for="(item, index) in breadcrumbNodes" :key="item.id" @click="breadcrumbNodeClick(index)">
<span class="tree-breadcrumb-start" v-if="index == 0">@</span>
<span class="tree-breadcrumb-link">{{ item.name }}</span>
<span class="tree-breadcrumb-separator" v-if="index != breadcrumbNodes.length - 1">/</span>
</span>
</div>

<ul class="tree-list">
<li v-for="(item, index) in currentNodes" :key="item.id" @click="listclick2(index)">
{{ item.name }}
</li>
</ul>
</div>
</div>
</template>

参考链接

想看下视频网站的 只看TA 怎么实现的, 所以想抓包. 情况如下:

  • 优酷电脑没有, 手机有不收费
  • 腾讯电脑没有, 手机没有, 好像收费才看到
  • 爱奇艺电脑有

开始没看爱奇艺, 所以才抓的优酷包, 而优酷包有证书绑定, 抓不到包.
爱奇艺的格式是

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
{
"id": 0,
"stars": ["200072305"],
"points": [
{
"startTime": 192,
"endTime": 233
},
{
"startTime": 321,
"endTime": 334
},
{
"startTime": 811,
"endTime": 1033
},
{
"startTime": 1248,
"endTime": 1405
},
{
"startTime": 1500,
"endTime": 1581
},
{
"startTime": 1621,
"endTime": 1741
}
],
"totalTime": 634
}

200072305 是演员的id. points 就是她在的视频段. 上面例子就是代表有6个片段包含该演员. startTime, endTime 就是片段的开始时间和结束时间. totalTime就是片段总的时长, 单位是秒, 所以在页面显示时长为10:34.

手机端

去github下载frida-server包, 需要下载手机相应的架构, 在手机端运行.
例如电脑上的安卓模拟器下载frida-server-16.0.2-android-x86_64.xz, android代表运行在安卓手机, x86_64代表模拟器是电脑端的64位. 解压后把文件推送到手机上.

1
2
# 推送文件
adb push frida-server-16.0.2-android-x86_64 /data/local/tmp

1
2
3
4
5
6
7
8
# 运行frida-server
adb shell
cd /data/local/tmp
# 修改权限, 手机需要root
su
chmod 777 frida-server-16.0.2-android-x86_64
# 运行
./frida-server-16.0.2-android-x86_64

电脑端

安装frida

1
pip install frida-tools

启动要查看的程序, 然后运行, 可以找到程序的包名

1
adb shell dumpsys window w |findstr \/ |findstr name=

hook文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
# hookssl.js
Java.perform(function() {

/*
hook list:
1.SSLcontext
2.okhttp
3.webview
4.XUtils
5.httpclientandroidlib
6.JSSE
7.network\_security\_config (android 7.0+)
8.Apache Http client (support partly)
9.OpenSSLSocketImpl
10.TrustKit
11.Cronet
*/

// Attempts to bypass SSL pinning implementations in a number of
// ways. These include implementing a new TrustManager that will
// accept any SSL certificate, overriding OkHTTP v3 check()
// method etc.
var X509TrustManager = Java.use('javax.net.ssl.X509TrustManager');
var HostnameVerifier = Java.use('javax.net.ssl.HostnameVerifier');
var SSLContext = Java.use('javax.net.ssl.SSLContext');
var quiet_output = false;

// Helper method to honor the quiet flag.

function quiet_send(data) {

if (quiet_output) {

return;
}

send(data)
}


// Implement a new TrustManager
// ref: https://gist.github.com/oleavr/3ca67a173ff7d207c6b8c3b0ca65a9d8
// Java.registerClass() is only supported on ART for now(201803). 所以android 4.4以下不兼容,4.4要切换成ART使用.
/*
06-07 16:15:38.541 27021-27073/mi.sslpinningdemo W/System.err: java.lang.IllegalArgumentException: Required method checkServerTrusted(X509Certificate[], String, String, String) missing
06-07 16:15:38.542 27021-27073/mi.sslpinningdemo W/System.err: at android.net.http.X509TrustManagerExtensions.<init>(X509TrustManagerExtensions.java:73)
at mi.ssl.MiPinningTrustManger.<init>(MiPinningTrustManger.java:61)
06-07 16:15:38.543 27021-27073/mi.sslpinningdemo W/System.err: at mi.sslpinningdemo.OkHttpUtil.getSecPinningClient(OkHttpUtil.java:112)
at mi.sslpinningdemo.OkHttpUtil.get(OkHttpUtil.java:62)
at mi.sslpinningdemo.MainActivity$1$1.run(MainActivity.java:36)
*/
var X509Certificate = Java.use("java.security.cert.X509Certificate");
var TrustManager;
try {
TrustManager = Java.registerClass({
name: 'org.wooyun.TrustManager',
implements: [X509TrustManager],
methods: {
checkClientTrusted: function(chain, authType) {},
checkServerTrusted: function(chain, authType) {},
getAcceptedIssuers: function() {
// var certs = [X509Certificate.$new()];
// return certs;
return [];
}
}
});
} catch (e) {
quiet_send("registerClass from X509TrustManager >>>>>>>> " + e.message);
}





// Prepare the TrustManagers array to pass to SSLContext.init()
var TrustManagers = [TrustManager.$new()];

try {
// Prepare a Empty SSLFactory
var TLS_SSLContext = SSLContext.getInstance("TLS");
TLS_SSLContext.init(null, TrustManagers, null);
var EmptySSLFactory = TLS_SSLContext.getSocketFactory();
} catch (e) {
quiet_send(e.message);
}

send('Custom, Empty TrustManager ready');

// Get a handle on the init() on the SSLContext class
var SSLContext_init = SSLContext.init.overload(
'[Ljavax.net.ssl.KeyManager;', '[Ljavax.net.ssl.TrustManager;', 'java.security.SecureRandom');

// Override the init method, specifying our new TrustManager
SSLContext_init.implementation = function(keyManager, trustManager, secureRandom) {

quiet_send('Overriding SSLContext.init() with the custom TrustManager');

SSLContext_init.call(this, null, TrustManagers, null);
};

/*** okhttp3.x unpinning ***/


// Wrap the logic in a try/catch as not all applications will have
// okhttp as part of the app.
try {

var CertificatePinner = Java.use('okhttp3.CertificatePinner');

quiet_send('OkHTTP 3.x Found');

CertificatePinner.check.overload('java.lang.String', 'java.util.List').implementation = function() {

quiet_send('OkHTTP 3.x check() called. Not throwing an exception.');
}

} catch (err) {

// If we dont have a ClassNotFoundException exception, raise the
// problem encountered.
if (err.message.indexOf('ClassNotFoundException') === 0) {

throw new Error(err);
}
}

// Appcelerator Titanium PinningTrustManager

// Wrap the logic in a try/catch as not all applications will have
// appcelerator as part of the app.
try {

var PinningTrustManager = Java.use('appcelerator.https.PinningTrustManager');

send('Appcelerator Titanium Found');

PinningTrustManager.checkServerTrusted.implementation = function() {

quiet_send('Appcelerator checkServerTrusted() called. Not throwing an exception.');
}

} catch (err) {

// If we dont have a ClassNotFoundException exception, raise the
// problem encountered.
if (err.message.indexOf('ClassNotFoundException') === 0) {

throw new Error(err);
}
}

/*** okhttp unpinning ***/


try {
var OkHttpClient = Java.use("com.squareup.okhttp.OkHttpClient");
OkHttpClient.setCertificatePinner.implementation = function(certificatePinner) {
// do nothing
quiet_send("OkHttpClient.setCertificatePinner Called!");
return this;
};

// Invalidate the certificate pinnet checks (if "setCertificatePinner" was called before the previous invalidation)
var CertificatePinner = Java.use("com.squareup.okhttp.CertificatePinner");
CertificatePinner.check.overload('java.lang.String', '[Ljava.security.cert.Certificate;').implementation = function(p0, p1) {
// do nothing
quiet_send("okhttp Called! [Certificate]");
return;
};
CertificatePinner.check.overload('java.lang.String', 'java.util.List').implementation = function(p0, p1) {
// do nothing
quiet_send("okhttp Called! [List]");
return;
};
} catch (e) {
quiet_send("com.squareup.okhttp not found");
}

/*** WebView Hooks ***/

/* frameworks/base/core/java/android/webkit/WebViewClient.java */
/* public void onReceivedSslError(Webview, SslErrorHandler, SslError) */
var WebViewClient = Java.use("android.webkit.WebViewClient");

WebViewClient.onReceivedSslError.implementation = function(webView, sslErrorHandler, sslError) {
quiet_send("WebViewClient onReceivedSslError invoke");
//执行proceed方法
sslErrorHandler.proceed();
return;
};

WebViewClient.onReceivedError.overload('android.webkit.WebView', 'int', 'java.lang.String', 'java.lang.String').implementation = function(a, b, c, d) {
quiet_send("WebViewClient onReceivedError invoked");
return;
};

WebViewClient.onReceivedError.overload('android.webkit.WebView', 'android.webkit.WebResourceRequest', 'android.webkit.WebResourceError').implementation = function() {
quiet_send("WebViewClient onReceivedError invoked");
return;
};

/*** JSSE Hooks ***/

/* libcore/luni/src/main/java/javax/net/ssl/TrustManagerFactory.java */
/* public final TrustManager[] getTrustManager() */
/* TrustManagerFactory.getTrustManagers maybe cause X509TrustManagerExtensions error */
// var TrustManagerFactory = Java.use("javax.net.ssl.TrustManagerFactory");
// TrustManagerFactory.getTrustManagers.implementation = function(){
// quiet_send("TrustManagerFactory getTrustManagers invoked");
// return TrustManagers;
// }

var HttpsURLConnection = Java.use("javax.net.ssl.HttpsURLConnection");
/* libcore/luni/src/main/java/javax/net/ssl/HttpsURLConnection.java */
/* public void setDefaultHostnameVerifier(HostnameVerifier) */
HttpsURLConnection.setDefaultHostnameVerifier.implementation = function(hostnameVerifier) {
quiet_send("HttpsURLConnection.setDefaultHostnameVerifier invoked");
return null;
};
/* libcore/luni/src/main/java/javax/net/ssl/HttpsURLConnection.java */
/* public void setSSLSocketFactory(SSLSocketFactory) */
HttpsURLConnection.setSSLSocketFactory.implementation = function(SSLSocketFactory) {
quiet_send("HttpsURLConnection.setSSLSocketFactory invoked");
return null;
};
/* libcore/luni/src/main/java/javax/net/ssl/HttpsURLConnection.java */
/* public void setHostnameVerifier(HostnameVerifier) */
HttpsURLConnection.setHostnameVerifier.implementation = function(hostnameVerifier) {
quiet_send("HttpsURLConnection.setHostnameVerifier invoked");
return null;
};

/*** Xutils3.x hooks ***/
//Implement a new HostnameVerifier
var TrustHostnameVerifier;
try {
TrustHostnameVerifier = Java.registerClass({
name: 'org.wooyun.TrustHostnameVerifier',
implements: [HostnameVerifier],
method: {
verify: function(hostname, session) {
return true;
}
}
});

} catch (e) {
//java.lang.ClassNotFoundException: Didn't find class "org.wooyun.TrustHostnameVerifier"
quiet_send("registerClass from hostnameVerifier >>>>>>>> " + e.message);
}

try {
var RequestParams = Java.use('org.xutils.http.RequestParams');
RequestParams.setSslSocketFactory.implementation = function(sslSocketFactory) {
sslSocketFactory = EmptySSLFactory;
return null;
}

RequestParams.setHostnameVerifier.implementation = function(hostnameVerifier) {
hostnameVerifier = TrustHostnameVerifier.$new();
return null;
}

} catch (e) {
quiet_send("Xutils hooks not Found");
}

/*** httpclientandroidlib Hooks ***/
try {
var AbstractVerifier = Java.use("ch.boye.httpclientandroidlib.conn.ssl.AbstractVerifier");
AbstractVerifier.verify.overload('java.lang.String', '[Ljava.lang.String', '[Ljava.lang.String', 'boolean').implementation = function() {
quiet_send("httpclientandroidlib Hooks");
return null;
}
} catch (e) {
quiet_send("httpclientandroidlib Hooks not found");
}

/***
android 7.0+ network_security_config TrustManagerImpl hook
apache httpclient partly
***/
var TrustManagerImpl = Java.use("com.android.org.conscrypt.TrustManagerImpl");
// try {
// var Arrays = Java.use("java.util.Arrays");
// //apache http client pinning maybe baypass
// //https://github.com/google/conscrypt/blob/c88f9f55a523f128f0e4dace76a34724bfa1e88c/platform/src/main/java/org/conscrypt/TrustManagerImpl.java#471
// TrustManagerImpl.checkTrusted.implementation = function (chain, authType, session, parameters, authType) {
// quiet_send("TrustManagerImpl checkTrusted called");
// //Generics currently result in java.lang.Object
// return Arrays.asList(chain);
// }
//
// } catch (e) {
// quiet_send("TrustManagerImpl checkTrusted nout found");
// }

try {
// Android 7+ TrustManagerImpl
TrustManagerImpl.verifyChain.implementation = function(untrustedChain, trustAnchorChain, host, clientAuth, ocspData, tlsSctData) {
quiet_send("TrustManagerImpl verifyChain called");
// Skip all the logic and just return the chain again :P
//https://www.nccgroup.trust/uk/about-us/newsroom-and-events/blogs/2017/november/bypassing-androids-network-security-configuration/
// https://github.com/google/conscrypt/blob/c88f9f55a523f128f0e4dace76a34724bfa1e88c/platform/src/main/java/org/conscrypt/TrustManagerImpl.java#L650
return untrustedChain;
}
} catch (e) {
quiet_send("TrustManagerImpl verifyChain nout found below 7.0");
}
// OpenSSLSocketImpl
try {
var OpenSSLSocketImpl = Java.use('com.android.org.conscrypt.OpenSSLSocketImpl');
OpenSSLSocketImpl.verifyCertificateChain.implementation = function(certRefs, authMethod) {
quiet_send('OpenSSLSocketImpl.verifyCertificateChain');
}

quiet_send('OpenSSLSocketImpl pinning')
} catch (err) {
quiet_send('OpenSSLSocketImpl pinner not found');
}
// Trustkit
try {
var Activity = Java.use("com.datatheorem.android.trustkit.pinning.OkHostnameVerifier");
Activity.verify.overload('java.lang.String', 'javax.net.ssl.SSLSession').implementation = function(str) {
quiet_send('Trustkit.verify1: ' + str);
return true;
};
Activity.verify.overload('java.lang.String', 'java.security.cert.X509Certificate').implementation = function(str) {
quiet_send('Trustkit.verify2: ' + str);
return true;
};

quiet_send('Trustkit pinning')
} catch (err) {
quiet_send('Trustkit pinner not found')
}

try {
//cronet pinner hook
//weibo don't invoke

var netBuilder = Java.use("org.chromium.net.CronetEngine$Builder");

//https://developer.android.com/guide/topics/connectivity/cronet/reference/org/chromium/net/CronetEngine.Builder.html#enablePublicKeyPinningBypassForLocalTrustAnchors(boolean)
netBuilder.enablePublicKeyPinningBypassForLocalTrustAnchors.implementation = function(arg) {

//weibo not invoke
console.log("Enables or disables public key pinning bypass for local trust anchors = " + arg);

//true to enable the bypass, false to disable.
var ret = netBuilder.enablePublicKeyPinningBypassForLocalTrustAnchors.call(this, true);
return ret;
};

netBuilder.addPublicKeyPins.implementation = function(hostName, pinsSha256, includeSubdomains, expirationDate) {
console.log("cronet addPublicKeyPins hostName = " + hostName);

//var ret = netBuilder.addPublicKeyPins.call(this,hostName, pinsSha256,includeSubdomains, expirationDate);
//this 是调用 addPublicKeyPins 前的对象吗? Yes,CronetEngine.Builder
return this;
};

} catch (err) {
console.log('[-] Cronet pinner not found')
}
});

运行com.youku.phone 是x酷的包名, 改成你抓包的

1
frida -U -l hookssl.js -f com.youku.phone

在抓包软件已经可以看到之前抓不到的 https 包了.

参考链接

起因

在处理实时视频流分析中, 如果AI推理推理时间太长导致, 会导致视频流中断和延迟.
尝试情况:

  • opencv + rtsp + ffmpeg + 多线程. 大概几十到几百帧就中断了
  • opencv + rtsp + ffmpeg + 多线程 + 重连. 在重连阶段会出现没有图像
  • opencv + rtsp + ffmpeg + 多进程 + 图像帧队列. 会出现中断, 可能是多进程间传输图像太大. 而且由于传输数据, 导致从抓取到推理时间变长很多
  • opencv + rtsp + gstreamer + 多线程. 解决中断问题. 推理时间短的话, 图像延迟较少, 如果推理时间长, 会造成延迟很久

由于python GIL(Global Interpreter Lock) 全局锁的限制, 多线程无法利用多核, 在一个cpu上交替运行的.
导致推理和读取视频交替, 如果推理时间长, 那么读取视频的次数就变少了, 而视频的帧数是相对不变的, 这样就造成了视频图像延迟.
所以多线程无法从根本上解决延迟.

多进程可以利用多核, 这样推理和读取视频在不同进程, 就互不影响, 读取视频可以做到实时读取. 而进程之间的通信方式有管道, 队列, 共享内存, 信号量, 信号, socket, 其中大数据传输最快的就是共享内存.

python3.8开始支持共享内存multiprocessing.shared_memory. 经过测试, 多进程加共享内存, 可以解决中断和延迟问题.

shared_memory 学习

官方文档比较简单, 但也覆盖了基本使用. 通过搜该关键字实际使用的也比较少, 比较相关的开源项目是 blakeblackshear/frigate, 它是一个实时的本地IP摄像头对象检测的NVR(网络视频记录器).

这里说些文档没有的.
size 参数是字节长度. 比如保存的是图像, 图像的长是1920, 宽是1080, 彩色的, 是用opencv读取的, 那么 size = 1920 * 1080 * 3. 可以通过img.nbytes查看.

由于内存是一维的, 所以赋值的时候需要把三维的图像转换为一维, 例如img.flatten().

从内存里面取出来的也是一维, 需要转换为三维图片. 而且需要指定读取的字节长度, 因为创建的长度可能因为平台不同, 有的平台会限制最小单位, 所以实际创建的会等于或者大于指定的size.

1
2
# count是字节长度, shape是返回的形状
np.frombuffer(bytes(shm.buf[:]), dtype=np.uint8, count=count).reshape(shape)

文档没有说明windows和linux的区别.
在windwos上会出现的不一致情况:

  • 在windows上调用close(), 如果之前还没有其他进程获取该内存块名的话, 那么其他线程无法在获取. 有人说是没有继承父进程的追踪器, 导致每个进程有自己的追踪器, 所以子进程的追踪器发现close, 就释放的该内存名, 其他子进程就获取不到了. 由于追踪器没有提供文档, 也不想在深入了解了, 而且也想出来一个简单的解决办法, p1进程先不close, 把共享内存名通过队列a传输到p2进程, 等p2进程处理完close, 在通过队列b传输内存名到p1进程, p1进程进行close.
  • unlink, 查看源码, 在windows上是不处理的.

坑2: windows 上内存泄漏

在windows上就算close了, 如果进程没有结束, 内存就没有释放掉, 造成内存一直增长.
这个bug 2020年在python3.8就发现, 并提出了解决办法,快2年多了都没处理, 可以看下面的参考链接. 目前只能通过 monkey-patch 来解决了, 通过引入下面的monkey_shared_memory() 方法.

1
2
3
4
5
6
7
import os
import sys


def monkey_shared_memory():
if os.name == "nt" and sys.version_info[:2] >= (3, 8):
from . import shared_memory

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
# shared_memory.py
"""
解决windows上shared-memory内存泄漏问题
https://bugs.python.org/issue40882
https://stackoverflow.com/questions/65968882/unlink-does-not-work-in-pythons-shared-memory-on-windows
"""
import ctypes
import ctypes.wintypes
import errno
import mmap
import multiprocessing
import multiprocessing.shared_memory
import os
import secrets
from multiprocessing.shared_memory import SharedMemory

if os.name == "nt":
import _winapi

_USE_POSIX = False
else:
import _posixshmem

_USE_POSIX = True


_O_CREX = os.O_CREAT | os.O_EXCL

# FreeBSD (and perhaps other BSDs) limit names to 14 characters.
_SHM_SAFE_NAME_LENGTH = 14

# Shared memory block name prefix
if _USE_POSIX:
_SHM_NAME_PREFIX = "/psm_"
else:
_SHM_NAME_PREFIX = "wnsm_"


def _make_filename():
"Create a random filename for the shared memory object."
# number of random bytes to use for name
nbytes = (_SHM_SAFE_NAME_LENGTH - len(_SHM_NAME_PREFIX)) // 2
assert nbytes >= 2, "_SHM_NAME_PREFIX too long"
name = _SHM_NAME_PREFIX + secrets.token_hex(nbytes)
assert len(name) <= _SHM_SAFE_NAME_LENGTH
return name


UnmapViewOfFile = ctypes.windll.kernel32.UnmapViewOfFile
UnmapViewOfFile.argtypes = (ctypes.wintypes.LPCVOID,)
UnmapViewOfFile.restype = ctypes.wintypes.BOOL


class _SharedMemory(SharedMemory):
def __init__(self, name=None, create=False, size=0):
if not size >= 0:
raise ValueError("'size' must be a positive integer")
if create:
self._flags = _O_CREX | os.O_RDWR
if size == 0:
raise ValueError("'size' must be a positive number different from zero")
if name is None and not self._flags & os.O_EXCL:
raise ValueError("'name' can only be None if create=True")

if _USE_POSIX:

# POSIX Shared Memory

if name is None:
while True:
name = _make_filename()
try:
self._fd = _posixshmem.shm_open(
name, self._flags, mode=self._mode
)
except FileExistsError:
continue
self._name = name
break
else:
name = "/" + name if self._prepend_leading_slash else name
self._fd = _posixshmem.shm_open(name, self._flags, mode=self._mode)
self._name = name
try:
if create and size:
os.ftruncate(self._fd, size)
stats = os.fstat(self._fd)
size = stats.st_size
self._mmap = mmap.mmap(self._fd, size)
except OSError:
self.unlink()
raise

from .resource_tracker import register

register(self._name, "shared_memory")

else:

# Windows Named Shared Memory

if create:
while True:
temp_name = _make_filename() if name is None else name
# Create and reserve shared memory block with this name
# until it can be attached to by mmap.
h_map = _winapi.CreateFileMapping(
_winapi.INVALID_HANDLE_VALUE,
_winapi.NULL,
_winapi.PAGE_READWRITE,
(size >> 32) & 0xFFFFFFFF,
size & 0xFFFFFFFF,
temp_name,
)
try:
last_error_code = _winapi.GetLastError()
if last_error_code == _winapi.ERROR_ALREADY_EXISTS:
if name is not None:
raise FileExistsError(
errno.EEXIST,
os.strerror(errno.EEXIST),
name,
_winapi.ERROR_ALREADY_EXISTS,
)
else:
continue
self._mmap = mmap.mmap(-1, size, tagname=temp_name)
finally:
_winapi.CloseHandle(h_map)
self._name = temp_name
break

else:
self._name = name
# Dynamically determine the existing named shared memory
# block's size which is likely a multiple of mmap.PAGESIZE.
h_map = _winapi.OpenFileMapping(_winapi.FILE_MAP_READ, False, name)
try:
p_buf = _winapi.MapViewOfFile(h_map, _winapi.FILE_MAP_READ, 0, 0, 0)
finally:
_winapi.CloseHandle(h_map)

try:
size = _winapi.VirtualQuerySize(p_buf)
self._mmap = mmap.mmap(-1, size, tagname=name)
finally:
UnmapViewOfFile(p_buf)

self._size = size
self._buf = memoryview(self._mmap)


multiprocessing.shared_memory.SharedMemory = _SharedMemory

参考链接