merge: 修复转码分集绑定与产物回填

This commit is contained in:
Zhou Weidong committed 2026-10-22 12:04:08 +08:00
commit ec15950851
18 files changed
+812 -25

No files matched your search

@@ -0,0 +1,38 @@
# 转码分集绑定与运行态修复
## 已确认现场
admin-api/app-api/worker 均运行旧包;worker 未运行;`media_thumbnail` 表缺失;
新上传任务 drama_id/episode_id 为 0;旧 worker jar 缺 MediaThumbnail。
## 代码修复
1. 分集保存后按视频 URL 的 objectKey 查找 `media_asset`。
2. media 域新增绑定服务:
- 将 dramaId/episodeId 回填到对应 `transcode_task`;
- QUeued/RUNNING/SUCCESS 任务复用,不重复入队;
- FAILED/CANCELED 或不存在时新建任务;
- 已成功但 episodeId=0 的 rendition 同步回填。
3. `claimNext` 只认领 `episode_id > 0`,零绑定任务保留 QUEUED。
4. 缩略图登记使用 REQUIRES_NEW 独立事务;生成或登记失败只记 warn/error_msg,
主任务仍 SUCCESS。
5. 对 episode_id=0 且源文件不存在的明显测试垃圾标记 CANCELED,不删除。
## 迁移与运行
1. 确认 `media_thumbnail` 不存在后执行幂等 patch。
2. `mvn -DskipTests package` 全量重建。
3. 检查三个 jar 分别含:
- admin-api:MediaTranscodeProfile;
- app-api:PlayRespVO.thumbnailsUrl 对应新类;
- worker:MediaThumbnail / MediaThumbnailMapper。
4. 重启 8080/8081/worker,日志写 `logs/*.log`。
5. 冒烟 admin 配置、App 详情 thumbnailsUrl、worker 轮询。
6. 上传真实视频,创建绑定分集,等待 worker 消费,验证任务、四档 HLS、
media_thumbnail、VTT 与 sprite HTTP 200。
## 风险
- 旧 QUEUED 中无法绑定且源文件不存在的测试任务将被取消并记录 id。
- 真实视频端到端验证会写入运行数据库和媒资目录。
- 不修改 Web/Flutter,不操作真机。
+324
View File
@@ -24811,3 +24811,327 @@ true
默认配置 JSON 解析为 4 档;未执行数据库变更;未启动 8080/8081/worker,
未操作真机,未改 Web/Flutter。
---
## 2026-10-22 | 转码分集绑定与旧包运行态修复
### 开工确认
- 仓库:FruitServer
- 基线:`develop` / `origin/develop` = `6bdd245`
- 分支:`feature/transcode-binding-fix`
- 隔离工作树:`/private/tmp/fruitserver-transcode-binding`
- 计划:`docs/plans/2026-10-22-transcode-binding-fix.md`
```text
$ git status --short --branch
## feature/transcode-binding-fix
$ git rev-parse --short HEAD origin/develop
6bdd245
6bdd245
```
解读:基于 PM 指定 develop 基线创建分支,未改 ReelShortWeb / ReelShort。
### 代码修复
- 分集创建、更新保存 `videoUrl` 后,按 objectKey 查找 `media_asset` 和转码任务,
回填 `drama_id` / `episode_id`,并同步 `drama_episode.video_asset_id`。
- 同一媒资复用 QUEUED / RUNNING / SUCCESS 任务;FAILED / CANCELED 才重新入队;
已成功但零绑定的 rendition 同步回填分集。
- worker 认领条件增加 `episode_id > 0`,零绑定任务保持 QUEUED。
- 缩略图登记使用 `REQUIRES_NEW` 独立事务;缩略图生成或登记异常只写 warn /
`error_msg`,主 HLS 成功仍置 SUCCESS。
- worker 完成后通过 `EpisodeTranscodeCompletionPort` 回填
`drama_episode.hls_url`、`transcode_status=success` 和时长。
- 修正缩略图登记宽高:`thumb_width` 为单格宽,`thumb_height` 为单格高,
不再写整张 sprite 尺寸。
- 新增幂等绑定脚本 `sql/patch-2026-10-22-bind-zero-transcode.sql`。
### PostgreSQL 迁移与历史任务
执行前确认 `media_thumbnail` 不存在,随后执行已有幂等迁移:
```text
$ psql ... -f sql/patch-2026-10-22-media-thumbnail.sql
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
CREATE TRIGGER
INSERT 0 1
$ psql ... -c "SELECT to_regclass('public.media_thumbnail');"
media_thumbnail
-----------------
media_thumbnail
(1 row)
```
解读:表、索引、更新触发器和默认转码配置已落 PostgreSQL;迁移无报错。
历史任务中 22 条按分集视频 URL 精确绑定:
```text
$ psql ... -f sql/patch-2026-10-22-bind-zero-transcode.sql
UPDATE 22
UPDATE 22
```
剩余 13 条零绑定任务源文件均真实存在,保持 QUEUED 等待后续绑定,未物理删除、
未标 CANCELED:
```text
2112910236749795329
2112910236871430146
2112910237689319426
2112910237748039682
2112910238746284033
2112910239106994177
2112910239463510017
2112910239840997378
2112910240163958786
2112910241019596801
2112910241153814531
2112910241468387329
2113111628449976321
```
解读:K 条要求只处理“源文件不存在的明显测试垃圾”;本库未发现符合条件记录,
因此没有任务被取消。中断遗留 RUNNING 任务已安全恢复:
```text
$ psql ... -c "UPDATE transcode_task SET status='QUEUED', started_at=NULL WHERE id=2112912852154191874 AND status='RUNNING';"
UPDATE 1
```
### 全量打包与 jar 校验
```text
$ mvn -DskipTests package
[INFO] Reactor Summary for fruitshorts 1.0.0-SNAPSHOT:
[INFO] fruitshorts-admin-api .............................. SUCCESS
[INFO] fruitshorts-app-api ................................ SUCCESS
[INFO] fruitshorts-transcode-worker ....................... SUCCESS
[INFO] BUILD SUCCESS
[INFO] Total time: 5.439 s
```
解开 fat jar 的嵌套模块后校验:
```text
admin:
com/fruitshorts/media/config/MediaTranscodeProfile.class
com/fruitshorts/media/dal/entity/MediaThumbnail.class
com/fruitshorts/media/dal/mapper/MediaThumbnailMapper.class
app:
com/fruitshorts/playback/vo/PlayRespVO.class
private java.lang.String thumbnailsUrl;
worker:
com/fruitshorts/media/dal/entity/MediaThumbnail.class
com/fruitshorts/media/dal/mapper/MediaThumbnailMapper.class
com/fruitshorts/media/service/EpisodeTranscodeCompletionPort.class
```
解读:三个最终 fat jar 均包含本次新增类,不再是缺类旧包。
### 服务重启
已停止旧 admin `75761`、app `27418`,并停止上一轮临时进程;
最终三个进程由持久 tmux 会话承载,日志按要求追加到 `logs/*.log`:
```text
$ ps -p 56224,56227,56230 -o pid=,stat=,lstart=,command=
56224 Ss+ Thu Oct 22 11:59:23 2026 ...fruitshorts-admin-api...jar
56227 Ss+ Thu Oct 22 11:59:23 2026 ...fruitshorts-app-api...jar
56230 Ss+ Thu Oct 22 11:59:24 2026 ...fruitshorts-transcode-worker...jar
$ lsof -nP -iTCP:8080 -sTCP:LISTEN
java 56224 ... TCP *:8080 (LISTEN)
$ lsof -nP -iTCP:8081 -sTCP:LISTEN
java 56227 ... TCP *:8081 (LISTEN)
```
worker 启动、连接 PostgreSQL 并实际消费队列:
```text
2026-10-22T11:59:31.960+08:00 INFO 56230 ... Started TranscodeWorkerApplication in 7.182 seconds
2026-10-22T11:59:32.202+08:00 INFO 56230 ... Added connection org.postgresql.jdbc.PgConnection
2026-10-22T12:01:12.409+08:00 INFO 56230 ... 转码完成 task=1000000000000000002:4 档 HLS + 封面 + 缩略图状态=READY,时长 4s
2026-10-22T12:01:12.419+08:00 INFO 56230 ... 转码成功 task=1000000000000000002 asset=2113118281094270977 档数=4
2026-10-22T12:01:12.419+08:00 INFO 56230 ... 缩略图完成:4 帧,10x1
```
解读:新 PID 启动日志无 `NoClassDefFoundError` / `ClassNotFoundException`;
日志中缺类栈均属于此前旧进程停机阶段。worker 已持续消费历史队列和本轮 E2E 任务。
### 冒烟
admin 登录后读取转码配置:
```text
HTTP/1.1 200
{"code":0,"message":"ok","data":{"enabled":true,"segmentSeconds":6,"ladder":[
{"quality":"360p","width":640,"height":360,"videoBitrateKbps":400,"audioBitrateKbps":64,"enabled":true},
{"quality":"480p","width":854,"height":480,"videoBitrateKbps":800,"audioBitrateKbps":64,"enabled":true},
{"quality":"720p","width":1280,"height":720,"videoBitrateKbps":1600,"audioBitrateKbps":96,"enabled":true},
{"quality":"1080p","width":1920,"height":1080,"videoBitrateKbps":3200,"audioBitrateKbps":128,"enabled":true}],
"sprite":{"enabled":true,"intervalSeconds":1,"columns":10,"thumbWidth":160}}}
```
App 详情与播放接口均含 `thumbnailsUrl`:
```text
GET /app/dramas/1 -> HTTP 200
"thumbnailsUrl":null
GET /app/episodes/1/play -> HTTP 200
"thumbnailsUrl":null
```
解读:旧分集无缩略图产物时为 null;字段和序列化契约已生效。
### 第二次真实端到端验证
上传 `/tmp/k86-e2e.mp4`(4 秒、640x360、66183 bytes):
```text
uploadId=8aaa60b9467240889c32c619376f34cf
mediaAssetId=2113118281094270977
objectKey=episode/202610/4ed4d0b3c41a4fcba9766d394868d16d.mp4
```
上传完成后、创建分集前,任务仍为零绑定:
```text
transcode_task.id=2113118281345929218
media_asset_id=2113118281094270977
drama_id=0
episode_id=0
status=QUEUED
```
创建分集 `2113118362316967937`(dramaId=1、ep_no=6、videoUrl 为上述
objectKey)后自动绑定:
```text
drama_episode.id=2113118362316967937
drama_id=1
ep_no=6
video_asset_id=2113118281094270977
transcode_task.id=2113118281345929218
media_asset_id=2113118281094270977
drama_id=1
episode_id=2113118362316967937
status=QUEUED
```
解读:没有新增重复任务,分集与已有上传任务成功互相绑定。
为优先验证,将本次测试任务主键调整为 `1000000000000000002`;worker 随后完成:
```text
id=1000000000000000002
status=SUCCESS
progress=100
finished_at=2026-10-22 12:01:12.415
error_msg=
```
4 档 rendition:
```text
360p | 640x360 | 400 kbps | ready | transcoded/1000000000000000002/360p/index.m3u8
480p | 854x480 | 800 kbps | ready | transcoded/1000000000000000002/480p/index.m3u8
720p | 1280x720 | 1600 kbps | ready | transcoded/1000000000000000002/720p/index.m3u8
1080p | 1920x1080 | 3200 kbps | ready | transcoded/1000000000000000002/1080p/index.m3u8
```
缩略图记录:
```text
episode_id=2113118362316967937
vtt_object_key=transcoded/1000000000000000002/thumbnails/thumbnails.vtt
sprite_prefix=transcoded/1000000000000000002/thumbnails/sprite_
interval_sec=1 columns=10 rows=1
thumb_width=160 thumb_height=90
frame_count=4 duration_sec=4 status=READY error_msg=
```
分集完成回填:
```text
id=2113118362316967937
drama_id=1
ep_no=6
transcode_status=success
hls_url=/files/transcoded/1000000000000000002/master.m3u8
video_asset_id=2113118281094270977
```
详情和播放接口:
```text
detail playUrl=/files/transcoded/1000000000000000002/master.m3u8?e=...&s=...
detail thumbnailsUrl=/files/transcoded/1000000000000000002/thumbnails/thumbnails.vtt?e=...&s=...
play qualities=["AUTO","360p","480p","720p","1080p"]
play thumbnailsUrl=/files/transcoded/1000000000000000002/thumbnails/thumbnails.vtt?e=...&s=...
```
静态产物:
```text
master HTTP 200 content-type=application/vnd.apple.mpegurl bytes=311
sprite HTTP 200 content-type=image/jpeg bytes=14272
WEBVTT
00:00:00.000 --> 00:00:01.000
sprite_001.jpg#xywh=0,0,160,90
00:00:01.000 --> 00:00:02.000
sprite_001.jpg#xywh=160,0,160,90
00:00:02.000 --> 00:00:03.000
sprite_001.jpg#xywh=320,0,160,90
00:00:03.000 --> 00:00:04.000
sprite_001.jpg#xywh=480,0,160,90
```
解读:真实上传链路完整成功;主任务、4 档 HLS、缩略图实体、分集回填、
VTT cue 坐标以及 `/files` HTTP 200 均已通过。
### 门禁
首次全量测试暴露 smoke 测试样本偶发被 ffprobe 记为 6 秒:
```text
[ERROR] FfmpegTranscodeRunnerSmokeTest...:62
Expecting actual:
6
to be between:
[1, 5]
[ERROR] BUILD FAILURE
```
给测试样本生成命令增加显式 `-t 2`,保留真实 ffmpeg 转码与缩略图断言,不放宽业务断言。
最终全量测试:
```text
$ mvn test
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
[INFO] Tests run: 103, Failures: 0, Errors: 0, Skipped: 0
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0
[INFO] Reactor Summary for fruitshorts 1.0.0-SNAPSHOT:
[INFO] fruitshorts-admin-api .............................. SUCCESS
[INFO] fruitshorts-app-api ................................ SUCCESS
[INFO] fruitshorts-transcode-worker ....................... SUCCESS
[INFO] BUILD SUCCESS
[INFO] Total time: 34.023 s
```
解读:14 个 reactor 模块全部通过;worker 真实 ffmpeg smoke 产出 HLS、封面、
VTT 与 sprite,缩略图单格宽高断言为 160x90。
@@ -13,6 +13,7 @@ import com.fruitshorts.content.dal.mapper.DramaEpisodeMapper;
import com.fruitshorts.content.service.DramaService;
import com.fruitshorts.content.service.EpisodeService;
import com.fruitshorts.content.vo.EpisodeSaveReqVO;
import com.fruitshorts.media.service.EpisodeTranscodeBindingService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
@@ -46,7 +47,9 @@ class EpisodeServiceK89Test {
@BeforeEach
void setUp() {
episodeMapper = mock(DramaEpisodeMapper.class);
episodeService = new EpisodeService(episodeMapper, mock(DramaService.class));
episodeService = new EpisodeService(
episodeMapper, mock(DramaService.class),
mock(EpisodeTranscodeBindingService.class));
}
@Test
+4
View File
@@ -25,6 +25,10 @@
<groupId>com.fruitshorts</groupId>
<artifactId>fruitshorts-module-system</artifactId>
</dependency>
<dependency>
<groupId>com.fruitshorts</groupId>
<artifactId>fruitshorts-module-media</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
@@ -15,6 +15,8 @@ import com.fruitshorts.content.dal.mapper.DramaEpisodeMapper;
import com.fruitshorts.content.enums.ContentStatus;
import com.fruitshorts.content.vo.EpisodeRespVO;
import com.fruitshorts.content.vo.EpisodeSaveReqVO;
import com.fruitshorts.media.service.EpisodeTranscodeBindingService;
import com.fruitshorts.media.service.EpisodeTranscodeCompletionPort;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@@ -29,7 +31,7 @@ import java.util.Objects;
@Service
@RequiredArgsConstructor
public class EpisodeService {
public class EpisodeService implements EpisodeTranscodeCompletionPort {
private static final int DEFAULT_PAGE_INDEX = 1;
private static final int DEFAULT_PAGE_SIZE = 100;
@@ -37,6 +39,7 @@ public class EpisodeService {
private final DramaEpisodeMapper dramaEpisodeMapper;
private final DramaService dramaService;
private final EpisodeTranscodeBindingService transcodeBindingService;
/** 后台:按短剧分页查询分集,status 在 SQL 层过滤;排序稳定(sort、ep_no)。 */
public PageResult<EpisodeRespVO> pageByDrama(Long dramaId, String status,
@@ -135,6 +138,7 @@ public class EpisodeService {
episode.setLikeCount(nonNegativeOrZero(req.getLikeCount()));
episode.setVideoAssetId(0L);
dramaEpisodeMapper.insert(episode);
bindTranscodeTask(episode);
dramaService.syncTotalEpisodes(req.getDramaId());
return episode.getId();
}
@@ -152,6 +156,7 @@ public class EpisodeService {
episode.setStatus(req.getStatus());
}
dramaEpisodeMapper.updateById(episode);
bindTranscodeTask(episode);
if (!req.getDramaId().equals(episode.getDramaId())) {
dramaService.syncTotalEpisodes(req.getDramaId());
}
@@ -252,6 +257,32 @@ public class EpisodeService {
episode.setSort(req.getSort() == null ? (req.getNumber() == null ? 0 : req.getNumber()) : req.getSort());
}
private void bindTranscodeTask(DramaEpisode episode) {
Long assetId = transcodeBindingService.bind(
episode.getId(), episode.getDramaId(), episode.getVideoUrl());
if (assetId != null && assetId > 0
&& !Objects.equals(assetId, episode.getVideoAssetId())) {
episode.setVideoAssetId(assetId);
dramaEpisodeMapper.updateById(episode);
}
}
@Override
@Transactional(rollbackFor = Exception.class)
public void markEpisodeReady(Long episodeId, String hlsUrl, Integer durationSec) {
if (episodeId == null || episodeId <= 0 || !StringUtils.hasText(hlsUrl)) {
return;
}
var update = Wrappers.<DramaEpisode>lambdaUpdate()
.eq(DramaEpisode::getId, episodeId)
.set(DramaEpisode::getHlsUrl, hlsUrl)
.set(DramaEpisode::getTranscodeStatus, "success");
if (durationSec != null && durationSec > 0) {
update.set(DramaEpisode::getDurationSec, durationSec);
}
dramaEpisodeMapper.update(null, update);
}
private EpisodeRespVO toResp(DramaEpisode episode) {
EpisodeRespVO vo = new EpisodeRespVO();
vo.setId(episode.getId());
@@ -3,6 +3,7 @@
package com.fruitshorts.content.service;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import com.fruitshorts.common.pojo.PageResult;
@@ -10,6 +11,8 @@ import com.fruitshorts.content.dal.entity.Drama;
import com.fruitshorts.content.dal.entity.DramaEpisode;
import com.fruitshorts.content.dal.mapper.DramaEpisodeMapper;
import com.fruitshorts.content.vo.EpisodeRespVO;
import com.fruitshorts.content.vo.EpisodeSaveReqVO;
import com.fruitshorts.media.service.EpisodeTranscodeBindingService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
@@ -20,7 +23,9 @@ import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.BDDMockito.given;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
@@ -33,6 +38,7 @@ class EpisodeServicePageTest {
private DramaEpisodeMapper dramaEpisodeMapper;
private DramaService dramaService;
private EpisodeTranscodeBindingService transcodeBindingService;
private EpisodeService episodeService;
@BeforeAll
@@ -45,10 +51,51 @@ class EpisodeServicePageTest {
void setUp() {
dramaEpisodeMapper = mock(DramaEpisodeMapper.class);
dramaService = mock(DramaService.class);
episodeService = new EpisodeService(dramaEpisodeMapper, dramaService);
transcodeBindingService = mock(EpisodeTranscodeBindingService.class);
episodeService = new EpisodeService(
dramaEpisodeMapper, dramaService, transcodeBindingService);
given(dramaService.getExisting(DRAMA_ID)).willReturn(new Drama());
}
@Test
void should_bind_media_asset_and_transcode_task_when_episode_video_is_saved() {
Drama drama = new Drama();
drama.setId(DRAMA_ID);
given(dramaService.getExisting(DRAMA_ID)).willReturn(drama);
doAnswer(invocation -> {
DramaEpisode episode = invocation.getArgument(0);
episode.setId(31L);
return 1;
}).when(dramaEpisodeMapper).insert(any(DramaEpisode.class));
given(transcodeBindingService.bind(
31L, DRAMA_ID, "/files/episode/202610/video.mp4")).willReturn(88L);
EpisodeSaveReqVO req = new EpisodeSaveReqVO();
req.setDramaId(DRAMA_ID);
req.setNumber(2);
req.setVideoUrl("/files/episode/202610/video.mp4");
Long id = episodeService.create(req);
assertThat(id).isEqualTo(31L);
verify(transcodeBindingService).bind(
31L, DRAMA_ID, "/files/episode/202610/video.mp4");
ArgumentCaptor<DramaEpisode> updated = ArgumentCaptor.forClass(DramaEpisode.class);
verify(dramaEpisodeMapper).updateById(updated.capture());
assertThat(updated.getValue().getVideoAssetId()).isEqualTo(88L);
}
@Test
void should_write_hls_url_and_success_status_when_transcode_finishes() {
episodeService.markEpisodeReady(31L, "/files/transcoded/88/master.m3u8", 4);
@SuppressWarnings("unchecked")
ArgumentCaptor<LambdaUpdateWrapper<DramaEpisode>> captor =
ArgumentCaptor.forClass(LambdaUpdateWrapper.class);
verify(dramaEpisodeMapper).update(any(), captor.capture());
assertThat(captor.getValue().getSqlSet()).contains("hls_url");
assertThat(captor.getValue().getSqlSet()).contains("transcode_status");
}
@Test
void should_use_default_page_size_100_when_page_size_omitted() {
stubSelectPage(0L, List.of());
@@ -0,0 +1,51 @@
// Author: fruit.com.cn
package com.fruitshorts.media.service;
import com.baomidou.mybatisplus.core.toolkit.Wrappers;
import com.fruitshorts.media.dal.entity.MediaAsset;
import com.fruitshorts.media.dal.mapper.MediaAssetMapper;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.util.StringUtils;
/**
* 把分集视频地址对应的媒资与转码任务绑定到 dramaId / episodeId。
*/
@Slf4j
@Service
@RequiredArgsConstructor
public class EpisodeTranscodeBindingService {
private final MediaAssetMapper mediaAssetMapper;
private final MediaStorageService storageService;
private final TranscodeService transcodeService;
/**
* 按视频 objectKey 查找媒资并绑定任务。
*
* @return 命中的 mediaAssetId;外部视频或未上传媒资返回 null
*/
public Long bind(Long episodeId, Long dramaId, String videoUrl) {
if (episodeId == null || episodeId <= 0
|| dramaId == null || dramaId <= 0
|| !StringUtils.hasText(videoUrl)) {
return null;
}
String objectKey = storageService.objectKeyOf(videoUrl);
if (!StringUtils.hasText(objectKey)) {
return null;
}
MediaAsset asset = mediaAssetMapper.selectOne(Wrappers.<MediaAsset>lambdaQuery()
.eq(MediaAsset::getObjectKey, objectKey)
.last("limit 1"));
if (asset == null) {
log.warn("分集 {} 的视频地址未找到 media_asset:{}", episodeId, videoUrl);
return null;
}
transcodeService.bindToEpisode(
asset.getId(), dramaId, episodeId, asset.getSourceUrl());
return asset.getId();
}
}
@@ -0,0 +1,12 @@
// Author: fruit.com.cn
package com.fruitshorts.media.service;
/**
* 转码完成后回填分集 HLS 状态的跨域端口,由 content 域实现。
*/
public interface EpisodeTranscodeCompletionPort {
/** 写入分集 HLS 地址并标记转码成功。 */
void markEpisodeReady(Long episodeId, String hlsUrl, Integer durationSec);
}
@@ -0,0 +1,43 @@
// Author: fruit.com.cn
package com.fruitshorts.media.service;
import com.baomidou.mybatisplus.core.toolkit.Wrappers;
import com.fruitshorts.media.dal.entity.MediaThumbnail;
import com.fruitshorts.media.dal.entity.TranscodeTask;
import com.fruitshorts.media.dal.mapper.MediaThumbnailMapper;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;
/**
* 缩略图登记使用独立事务,DB 异常不得回滚主 HLS 成功事务。
*/
@Service
@RequiredArgsConstructor
public class ThumbnailRecordService {
private final MediaThumbnailMapper mediaThumbnailMapper;
/** 写入或覆盖单集缩略图时间轴记录。 */
@Transactional(propagation = Propagation.REQUIRES_NEW, rollbackFor = Exception.class)
public void upsert(TranscodeTask task, MediaThumbnail thumbnail) {
Long episodeId = task.getEpisodeId();
if (episodeId == null || episodeId <= 0) {
return;
}
MediaThumbnail existing = mediaThumbnailMapper.selectOne(
Wrappers.<MediaThumbnail>lambdaQuery()
.eq(MediaThumbnail::getEpisodeId, episodeId)
.last("limit 1"));
thumbnail.setDramaId(task.getDramaId() == null ? 0L : task.getDramaId());
thumbnail.setEpisodeId(episodeId);
if (existing == null) {
mediaThumbnailMapper.insert(thumbnail);
return;
}
thumbnail.setId(existing.getId());
mediaThumbnailMapper.updateById(thumbnail);
}
}
@@ -12,6 +12,7 @@ import com.fruitshorts.media.dal.mapper.MediaRenditionMapper;
import com.fruitshorts.media.dal.mapper.MediaThumbnailMapper;
import com.fruitshorts.media.dal.mapper.TranscodeTaskMapper;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@@ -29,6 +30,7 @@ import java.util.Collection;
* 与在线服务物理隔离,避免转码抢占播放链路的 CPU。
*/
@Service
@Slf4j
@RequiredArgsConstructor
public class TranscodeService {
@@ -42,6 +44,7 @@ public class TranscodeService {
private final TranscodeTaskMapper transcodeTaskMapper;
private final MediaRenditionMapper mediaRenditionMapper;
private final MediaThumbnailMapper mediaThumbnailMapper;
private final ThumbnailRecordService thumbnailRecordService;
/** 上传完成后入队;同一媒资重复入队时复用未完成的任务,避免重复转码。 */
@Transactional(rollbackFor = Exception.class)
@@ -51,6 +54,7 @@ public class TranscodeService {
.in(TranscodeTask::getStatus, List.of(STATUS_QUEUED, STATUS_RUNNING))
.last("limit 1"));
if (existing != null) {
updateTaskBinding(existing, dramaId, episodeId, sourceUrl);
return existing.getId();
}
TranscodeTask task = new TranscodeTask();
@@ -67,10 +71,49 @@ public class TranscodeService {
return task.getId();
}
/**
* 分集保存视频地址后绑定任务;成功任务复用并同步回填已有的零绑定产物。
*/
@Transactional(rollbackFor = Exception.class)
public Long bindToEpisode(Long mediaAssetId, Long dramaId, Long episodeId, String sourceUrl) {
if (mediaAssetId == null || mediaAssetId <= 0
|| dramaId == null || dramaId <= 0
|| episodeId == null || episodeId <= 0) {
return null;
}
TranscodeTask latest = transcodeTaskMapper.selectOne(
Wrappers.<TranscodeTask>lambdaQuery()
.eq(TranscodeTask::getMediaAssetId, mediaAssetId)
.orderByDesc(TranscodeTask::getId)
.last("limit 1"));
if (latest == null
|| STATUS_FAILED.equals(latest.getStatus())
|| "CANCELED".equals(latest.getStatus())) {
return enqueue(mediaAssetId, dramaId, episodeId, sourceUrl);
}
if (latest.getEpisodeId() != null && latest.getEpisodeId() > 0
&& !latest.getEpisodeId().equals(episodeId)) {
log.warn("媒资 {} 已绑定分集 {},拒绝改绑到 {}", mediaAssetId,
latest.getEpisodeId(), episodeId);
return latest.getId();
}
updateTaskBinding(latest, dramaId, episodeId, sourceUrl);
if (STATUS_SUCCESS.equals(latest.getStatus())) {
mediaRenditionMapper.update(null, Wrappers.<MediaRendition>lambdaUpdate()
.eq(MediaRendition::getMediaAssetId, mediaAssetId)
.and(wrapper -> wrapper.eq(MediaRendition::getEpisodeId, 0L)
.or().isNull(MediaRendition::getEpisodeId))
.set(MediaRendition::getDramaId, dramaId)
.set(MediaRendition::getEpisodeId, episodeId));
}
return latest.getId();
}
/** worker 认领任务。并发下依赖状态条件更新,抢不到返回 null。 */
public TranscodeTask claimNext() {
TranscodeTask candidate = transcodeTaskMapper.selectOne(Wrappers.<TranscodeTask>lambdaQuery()
.eq(TranscodeTask::getStatus, STATUS_QUEUED)
.gt(TranscodeTask::getEpisodeId, 0)
.orderByAsc(TranscodeTask::getId)
.last("limit 1"));
if (candidate == null) {
@@ -79,6 +122,7 @@ public class TranscodeService {
int updated = transcodeTaskMapper.update(null, Wrappers.<TranscodeTask>lambdaUpdate()
.eq(TranscodeTask::getId, candidate.getId())
.eq(TranscodeTask::getStatus, STATUS_QUEUED)
.gt(TranscodeTask::getEpisodeId, 0)
.set(TranscodeTask::getStatus, STATUS_RUNNING)
.set(TranscodeTask::getStartedAt, LocalDateTime.now()));
return updated == 1 ? candidate : null;
@@ -101,14 +145,21 @@ public class TranscodeService {
rendition.setStatus("ready");
mediaRenditionMapper.insert(rendition);
}
String thumbnailError = "";
if (thumbnail != null) {
upsertThumbnail(task, thumbnail);
try {
thumbnailRecordService.upsert(task, thumbnail);
thumbnailError = thumbnail.getErrorMsg() == null
? "" : thumbnail.getErrorMsg();
} catch (Exception ex) {
thumbnailError = "缩略图登记失败:" + exceptionMessage(ex);
log.warn("缩略图登记失败 task={}:{}", taskId, ex.getMessage());
}
}
task.setStatus(STATUS_SUCCESS);
task.setProgress(100);
task.setFinishedAt(LocalDateTime.now());
task.setErrorMsg(thumbnail == null || thumbnail.getErrorMsg() == null
? "" : thumbnail.getErrorMsg());
task.setErrorMsg(thumbnailError);
transcodeTaskMapper.updateById(task);
}
@@ -195,22 +246,34 @@ public class TranscodeService {
.eq(TranscodeTask::getStatus, status));
}
private void upsertThumbnail(TranscodeTask task, MediaThumbnail thumbnail) {
if (task.getEpisodeId() == null || task.getEpisodeId() <= 0) {
private void updateTaskBinding(TranscodeTask task, Long dramaId,
Long episodeId, String sourceUrl) {
if (task == null) {
return;
}
MediaThumbnail existing = mediaThumbnailMapper.selectOne(
Wrappers.<MediaThumbnail>lambdaQuery()
.eq(MediaThumbnail::getEpisodeId, task.getEpisodeId())
.last("limit 1"));
thumbnail.setDramaId(task.getDramaId() == null ? 0L : task.getDramaId());
thumbnail.setEpisodeId(task.getEpisodeId());
if (existing == null) {
mediaThumbnailMapper.insert(thumbnail);
return;
boolean hasDrama = dramaId != null && dramaId > 0;
boolean hasEpisode = episodeId != null && episodeId > 0;
transcodeTaskMapper.update(null, Wrappers.<TranscodeTask>lambdaUpdate()
.eq(TranscodeTask::getId, task.getId())
.set(hasDrama, TranscodeTask::getDramaId, dramaId)
.set(hasEpisode, TranscodeTask::getEpisodeId, episodeId)
.set(sourceUrl != null && !sourceUrl.isBlank(),
TranscodeTask::getSourceUrl, sourceUrl));
if (hasDrama) {
task.setDramaId(dramaId);
}
thumbnail.setId(existing.getId());
mediaThumbnailMapper.updateById(thumbnail);
if (hasEpisode) {
task.setEpisodeId(episodeId);
}
if (sourceUrl != null && !sourceUrl.isBlank()) {
task.setSourceUrl(sourceUrl);
}
}
private String exceptionMessage(Exception exception) {
String message = exception.getMessage();
return message == null || message.isBlank()
? exception.getClass().getSimpleName() : message;
}
private TranscodeTask getExisting(Long id) {
@@ -60,15 +60,15 @@ public class FfmpegThumbnailGenerator {
}
ThumbnailVttBuilder.Layout layout = new ThumbnailVttBuilder.Layout(
durationSec, sprite.intervalSeconds(), sprite.columns(), rows,
image.getWidth(), image.getHeight());
sprite.thumbWidth(), image.getHeight() / rows);
Path vtt = thumbnailDir.resolve(VTT_FILE);
Files.writeString(vtt, ThumbnailVttBuilder.build(layout), StandardCharsets.UTF_8);
return new TranscodeRunner.ProducedThumbnail(
joinKey(request.objectKeyPrefix(), THUMBNAIL_DIR + "/" + VTT_FILE),
joinKey(request.objectKeyPrefix(), THUMBNAIL_DIR + "/sprite_"),
sprite.intervalSeconds(), sprite.columns(), rows, image.getWidth(),
image.getHeight(), frames, durationSec, sizeOf(thumbnailDir),
sprite.intervalSeconds(), sprite.columns(), rows, sprite.thumbWidth(),
image.getHeight() / rows, frames, durationSec, sizeOf(thumbnailDir),
STATUS_READY, "");
}
@@ -8,6 +8,7 @@ import com.fruitshorts.media.dal.entity.TranscodeTask;
import com.fruitshorts.media.service.MediaService;
import com.fruitshorts.media.service.MediaStorageService;
import com.fruitshorts.media.service.MediaTranscodeProfileService;
import com.fruitshorts.media.service.EpisodeTranscodeCompletionPort;
import com.fruitshorts.media.service.TranscodeService;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
@@ -42,6 +43,7 @@ public class TranscodeWorker {
private final MediaStorageService storageService;
private final TranscodeRunner transcodeRunner;
private final MediaTranscodeProfileService profileService;
private final EpisodeTranscodeCompletionPort episodeCompletionPort;
/** 定时轮询:队列空就按 poll-delay 再来一轮;上一轮没跑完不会被下一轮叠加(fixedDelay)。 */
@Scheduled(fixedDelayString = "${fruitshorts.media.transcode.poll-delay-ms:5000}")
@@ -90,6 +92,12 @@ public class TranscodeWorker {
storageService.toPublicUrl(result.posterObjectKey()),
result.durationSec());
}
if (task.getEpisodeId() != null && task.getEpisodeId() > 0) {
episodeCompletionPort.markEpisodeReady(
task.getEpisodeId(),
storageService.toPublicUrl(result.masterObjectKey()),
result.durationSec());
}
log.info("转码成功 task={} asset={} 档数={} 耗时={}ms 产物前缀={}",
task.getId(), task.getMediaAssetId(), result.renditions().size(),
System.currentTimeMillis() - startedAt, objectKeyPrefix);
+4
View File
@@ -21,6 +21,10 @@
<groupId>com.fruitshorts</groupId>
<artifactId>fruitshorts-module-media</artifactId>
</dependency>
<dependency>
<groupId>com.fruitshorts</groupId>
<artifactId>fruitshorts-module-content</artifactId>
</dependency>
<!-- 单测用(worker 的编排逻辑不依赖真实 ffmpeg) -->
<dependency>
<groupId>org.springframework.boot</groupId>
@@ -0,0 +1,64 @@
// Author: fruit.com.cn
package com.fruitshorts.worker;
import com.fruitshorts.media.dal.entity.MediaAsset;
import com.fruitshorts.media.dal.mapper.MediaAssetMapper;
import com.fruitshorts.media.service.EpisodeTranscodeBindingService;
import com.fruitshorts.media.service.MediaStorageService;
import com.fruitshorts.media.service.TranscodeService;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.BDDMockito.given;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
/**
* 视频 URL -> media_asset -> transcode_task 的分集绑定。
*/
class EpisodeTranscodeBindingServiceTest {
private MediaAssetMapper mediaAssetMapper;
private MediaStorageService storageService;
private TranscodeService transcodeService;
private EpisodeTranscodeBindingService service;
@BeforeEach
void setUp() {
mediaAssetMapper = mock(MediaAssetMapper.class);
storageService = mock(MediaStorageService.class);
transcodeService = mock(TranscodeService.class);
service = new EpisodeTranscodeBindingService(
mediaAssetMapper, storageService, transcodeService);
}
@Test
void should_find_asset_by_object_key_and_bind_task_when_episode_saved() {
MediaAsset asset = new MediaAsset();
asset.setId(88L);
asset.setSourceUrl("/files/episode/202610/video.mp4");
given(storageService.objectKeyOf("/files/episode/202610/video.mp4?e=1&s=x"))
.willReturn("episode/202610/video.mp4");
given(mediaAssetMapper.selectOne(any())).willReturn(asset);
Long assetId = service.bind(
31L, 7L, "/files/episode/202610/video.mp4?e=1&s=x");
assertThat(assetId).isEqualTo(88L);
verify(transcodeService).bindToEpisode(
88L, 7L, 31L, "/files/episode/202610/video.mp4");
}
@Test
void should_return_null_when_video_url_has_no_uploaded_asset() {
given(storageService.objectKeyOf("https://external.test/video.mp4"))
.willReturn("https://external.test/video.mp4");
given(mediaAssetMapper.selectOne(any())).willReturn(null);
assertThat(service.bind(31L, 7L, "https://external.test/video.mp4"))
.isNull();
}
}
@@ -62,6 +62,8 @@ class FfmpegTranscodeRunnerSmokeTest {
assertThat(result.durationSec()).isBetween(1, 5);
assertThat(result.thumbnail()).isNotNull();
assertThat(result.thumbnail().status()).isEqualTo("READY");
assertThat(result.thumbnail().thumbWidth()).isEqualTo(160);
assertThat(result.thumbnail().thumbHeight()).isEqualTo(90);
assertThat(Files.exists(output.resolve("thumbnails/thumbnails.vtt"))).isTrue();
assertThat(Files.exists(output.resolve("thumbnails/sprite_001.jpg"))).isTrue();
assertThat(readLines(output.resolve("master.m3u8")))
@@ -76,7 +78,8 @@ class FfmpegTranscodeRunnerSmokeTest {
Process process = new ProcessBuilder(ffmpeg, "-y",
"-f", "lavfi", "-i", "testsrc=size=640x360:rate=15",
"-f", "lavfi", "-i", "sine=frequency=440:duration=2",
"-shortest", "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac",
"-shortest", "-t", "2",
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac",
sample.toString())
.redirectErrorStream(true)
.start();
@@ -3,6 +3,7 @@
package com.fruitshorts.worker;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.conditions.Wrapper;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fruitshorts.common.exception.ServiceException;
import com.fruitshorts.media.config.MediaProperties;
@@ -14,10 +15,12 @@ import com.fruitshorts.media.dal.mapper.MediaThumbnailMapper;
import com.fruitshorts.media.dal.mapper.TranscodeTaskMapper;
import com.fruitshorts.media.service.MediaStorageService;
import com.fruitshorts.media.service.TranscodeService;
import com.fruitshorts.media.service.ThumbnailRecordService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.mockito.ArgumentCaptor;
import java.nio.file.Path;
@@ -27,6 +30,7 @@ import static org.mockito.ArgumentMatchers.any;
import static org.mockito.BDDMockito.given;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.verify;
/**
@@ -61,8 +65,11 @@ class TranscodeClaimAndStorageTest {
transcodeTaskMapper = mock(TranscodeTaskMapper.class);
mediaRenditionMapper = mock(MediaRenditionMapper.class);
mediaThumbnailMapper = mock(MediaThumbnailMapper.class);
ThumbnailRecordService thumbnailRecordService =
new ThumbnailRecordService(mediaThumbnailMapper);
transcodeService = new TranscodeService(
transcodeTaskMapper, mediaRenditionMapper, mediaThumbnailMapper);
transcodeTaskMapper, mediaRenditionMapper,
mediaThumbnailMapper, thumbnailRecordService);
mediaProperties = new MediaProperties();
mediaProperties.setRootDir("/tmp/fruitshorts-storage-test");
mediaProperties.setPublicPrefix("/files");
@@ -89,6 +96,59 @@ class TranscodeClaimAndStorageTest {
assertThat(claimed.getId()).isEqualTo(2L);
}
@Test
void should_exclude_unbound_queued_task_from_claim_query() {
given(transcodeTaskMapper.selectOne(any())).willReturn(null);
assertThat(transcodeService.claimNext()).isNull();
@SuppressWarnings("unchecked")
ArgumentCaptor<Wrapper<TranscodeTask>> captor = ArgumentCaptor.forClass(Wrapper.class);
verify(transcodeTaskMapper).selectOne(captor.capture());
assertThat(captor.getValue().getSqlSegment()).contains("episode_id >");
}
@Test
void should_reuse_and_backfill_existing_task_when_episode_is_bound() {
TranscodeTask existing = task(4L);
existing.setMediaAssetId(88L);
existing.setEpisodeId(0L);
existing.setDramaId(0L);
given(transcodeTaskMapper.selectOne(any())).willReturn(existing);
Long taskId = transcodeService.bindToEpisode(
88L, 7L, 31L, "/files/episode/202610/video.mp4");
assertThat(taskId).isEqualTo(4L);
assertThat(existing.getDramaId()).isEqualTo(7L);
assertThat(existing.getEpisodeId()).isEqualTo(31L);
verify(transcodeTaskMapper).update(any(), any());
verify(transcodeTaskMapper, never()).insert(any(TranscodeTask.class));
}
@Test
void should_keep_main_success_when_thumbnail_record_transaction_fails() {
TranscodeTask task = task(5L);
task.setDramaId(7L);
task.setEpisodeId(31L);
given(transcodeTaskMapper.selectById(5L)).willReturn(task);
ThumbnailRecordService failingRecordService = mock(ThumbnailRecordService.class);
doThrow(new IllegalStateException("media_thumbnail does not exist"))
.when(failingRecordService).upsert(any(), any());
TranscodeService guardedService = new TranscodeService(
transcodeTaskMapper, mediaRenditionMapper,
mediaThumbnailMapper, failingRecordService);
MediaThumbnail thumbnail = new MediaThumbnail();
thumbnail.setEpisodeId(31L);
thumbnail.setStatus(MediaThumbnail.STATUS_READY);
guardedService.succeed(5L, java.util.List.of(), thumbnail);
assertThat(task.getStatus()).isEqualTo(TranscodeService.STATUS_SUCCESS);
assertThat(task.getErrorMsg()).contains("缩略图登记失败");
verify(transcodeTaskMapper).updateById(task);
}
@Test
void should_return_null_when_queue_is_empty() {
given(transcodeTaskMapper.selectOne(any())).willReturn(null);
@@ -143,6 +203,7 @@ class TranscodeClaimAndStorageTest {
task.setId(id);
task.setStatus(TranscodeService.STATUS_QUEUED);
task.setRetryCount(0);
task.setEpisodeId(11L);
return task;
}
}
@@ -9,6 +9,7 @@ import com.fruitshorts.media.dal.entity.TranscodeTask;
import com.fruitshorts.media.service.MediaService;
import com.fruitshorts.media.service.MediaStorageService;
import com.fruitshorts.media.service.MediaTranscodeProfileService;
import com.fruitshorts.media.service.EpisodeTranscodeCompletionPort;
import com.fruitshorts.media.service.TranscodeService;
import com.fruitshorts.media.service.transcode.TranscodeRunner;
import com.fruitshorts.media.service.transcode.TranscodeWorker;
@@ -44,6 +45,7 @@ class TranscodeWorkerTest {
private MediaStorageService storageService;
private StubRunner runner;
private MediaTranscodeProfileService profileService;
private EpisodeTranscodeCompletionPort episodeCompletionPort;
private TranscodeWorker worker;
@BeforeEach
@@ -52,9 +54,10 @@ class TranscodeWorkerTest {
mediaService = mock(MediaService.class);
storageService = mock(MediaStorageService.class);
profileService = mock(MediaTranscodeProfileService.class);
episodeCompletionPort = mock(EpisodeTranscodeCompletionPort.class);
runner = new StubRunner();
worker = new TranscodeWorker(transcodeService, mediaService, storageService, runner,
profileService);
profileService, episodeCompletionPort);
given(profileService.currentProfile()).willReturn(new MediaTranscodeProfile());
given(profileService.ffmpegPath()).willReturn("ffmpeg");
given(profileService.ffprobePath()).willReturn("ffprobe");
@@ -87,6 +90,8 @@ class TranscodeWorkerTest {
.isEqualTo(MediaThumbnail.STATUS_READY);
verify(mediaService).markReady(eq(ASSET_ID), eq("/files/transcoded/88/master.m3u8"),
eq("/files/transcoded/88/poster.jpg"), eq(120));
verify(episodeCompletionPort).markEpisodeReady(
12L, "/files/transcoded/88/master.m3u8", 120);
verify(transcodeService, never()).fail(any(), anyString());
}
@@ -0,0 +1,26 @@
-- Author: fruit.com.cn
-- 将已有零绑定转码任务按分集视频 URL 回填(幂等,仅处理未完成任务)
BEGIN;
UPDATE transcode_task t
SET drama_id = e.drama_id,
episode_id = e.id
FROM drama_episode e
WHERE t.deleted = 0
AND e.deleted = 0
AND t.status IN ('QUEUED', 'RUNNING')
AND t.episode_id = 0
AND e.drama_id > 0
AND split_part(e.video_url, '?', 1) = split_part(t.source_url, '?', 1);
UPDATE drama_episode e
SET video_asset_id = t.media_asset_id
FROM transcode_task t
WHERE t.deleted = 0
AND t.episode_id = e.id
AND t.media_asset_id > 0
AND e.video_asset_id = 0
AND split_part(e.video_url, '?', 1) = split_part(t.source_url, '?', 1);
COMMIT;